A strong answer picks one service and holds to it, because the marking rewards depth over breadth. Name specific screens and specific actions rather than the service in general. A gulf of execution appears where the user's intention has no corresponding control: a claim that must be started before eligibility can be checked, or a booking system that requires a reference number the user has never seen. A gulf of evaluation appears after action: a submission with no confirmation, or a status that says pending without saying since when. The signifier analysis should identify controls that are technically present and practically invisible. It should also separate the case where an affordance is genuinely absent from the case where it exists and is not signalled. The second half of the task separates a good answer from a fluent one. For each claim, name the evidence: a task analysis of the steps a real claimant takes, a think-aloud session with a few people new to the service, including someone who reads slowly and someone who works with a screen reader, a check of how the form behaves on a slow connection or an older phone, or a timed comparison of two routes through the flow. Keep observation and inference apart. The module's own signifier claim is a prediction from the distinction, not a measured result, so a critique that reports a fall in use has to show the measurement. An answer that asserts that a page is confusing, without naming how that could be shown, is doing criticism rather than analysis.

A strong answer holds one system in view, names specific screens or tasks, and argues the placement rather than asserting it. The mental-model account belongs in the tradition the module calls classical cognitivism, where mind and computer are coupled information processors and the aim is a general, predictive model of the user. GOMS sits in that tradition too, and the module places the cognitive side of the dispute on Vera and Simon, who argued that the claims of situated action could be met inside a symbolic framework. The situated account belongs in the paradigm the module calls phenomenologically situated, where interaction is meaning making and the setting carries part of the explanation. The two send you to different places. The cognitive account sends you to the user's model: you look for the control whose name or position does not match what the user expects, and you fix the mapping, the wording or the feedback. Its evidence is predictive, and a test can show you to be wrong. The situated account sends you to the setting: you watch what people do, what they improvise, and what they use to work out what the system is doing. Its evidence is observational, and it predicts nothing for a screen that has not been built. The exchange between Suchman and Vera and Simon in Cognitive Science in 1993 is the printed case of the disagreement. While it is unresolved, treat a mental model as a hypothesis about your users and test it where the work happens, and treat the setting as a source of design constraints rather than as noise. What would change your mind is a prediction that survives the setting, stated in advance so that the test is real.

A strong answer names one system, holds to it, and describes two or three specific interactions rather than the product in general, because the marking rewards a situated argument over a survey of features. State the division of labour before judging it. Licklider's test is whether the person keeps the goals, the hypotheses, the criteria and the evaluations while the machine does the work that can be routinised; Engelbart's is whether the language, the artefacts and the user's method are still treated as one system that changes together; the authorship test is whether the user can still make and revise an arrangement of their own, as Bush's reader makes a trail and Kay and Goldberg's owner writes in the medium. Apply each test to a named interaction. In a code editor, the machine completes a routine the developer decides to accept, which looks like augmentation, and the completions begin to set the shape of what is written, which looks like the routine taking the formulation. In a navigation app, the route is judged by the driver until the driver stops forming a route at all. A replacement claim is stronger where the user can no longer state the goal the machine is pursuing, and weaker where the user can inspect and undo the machine's contribution. The second half of the answer is what separates analysis from assertion. Say what evidence would settle it: a record of what you actually hand over in one week, drawn from your own files, edit history or journey history; a kept log of the occasions when you accepted a machine suggestion and the occasions when you changed the goal because of it, which is the comparison Licklider's account invites; a usability test with a few people new to the system, including someone who works with a screen reader and someone on a slow connection, because the authorship test fails first for those users; or a reading of the vendor's defaults, since a default the user never changes is evidence about who is holding the criterion. Keep observation and inference apart throughout. The unit's own sources are design proposals, histories and one 1968 system report, so they supply the tests and not the measurements; an answer that quotes a figure from them as if it settled a claim about your system has misread the evidence.

A strong answer separates the three constants and uses each for the claim it can carry. Capacity is the first move: Cowan's limit of about four chunks, observable only when recoding is prevented, means twelve unlabelled icons should not be treated as twelve equally available options, because the user can group them only if the grouping is visible. The redesign should therefore organise the actions into a small number of labelled groups, which uses Miller's recoding rather than fighting the limit. Pointing is the second move: Fitts' law makes movement time rise with the logarithm of the distance divided by target width, so shrinking the icons to fit a dense grid increases the time and the error rate, and the menu should be rearranged so common actions sit close to the thumb's resting position and are made larger. Choice is the third move: Hick's law makes reaction time rise with the logarithm of the number of alternatives, so the initial menu should offer few options and let the user reach the rest through one labelled group rather than by scanning all twelve. The load argument then settles the priority. Sweller's claim is that material which spends the user's capacity on itself is avoidable, so decorative icons that must be decoded cost more than they return; Wickens's claim is that interference depends on what a task shares, so the answer should not simply reduce the count of items but also check that the menu does not compete with the task the user is doing while banking. Say what evidence would show the fix worked. The natural design is a comparison against the present menu, with task completion time and error counts for the most common actions, using a sample large enough to see an effect, since Fitts' and Hick's laws give directional predictions even when the exact times differ. If the app can log real use, the time to complete each action before and after the change is the least intrusive measure; a small usability test with people new to the app gives more control but fewer people. This is where most answers fall short. A claim that the new menu is more usable is not evidence unless it names a measure, a comparison and a decision rule. Then state the result that would count against you. If a group of users is as fast and as accurate with twelve unlabelled icons as with the grouped menu, the capacity argument has carried nothing for them, and the redesign should be justified on other grounds or abandoned. If the redesign helps novices but slows experienced users who had learned the original positions, the answer must say which users the rule is for. A last move is to say who the redesign serves. The twelve actions are common to the app's users in general, and the users the present grid costs most are those whose sight is poor, whose hands are unsteady, or who are reading a second language: for them larger targets and the thumb's resting position matter more than the count of items, and labels matter more than icons. Then say what the evidence still does not cover. The comparison runs with whichever participants were available, so it speaks for them and not for the older or less practised user, and the claim should not reach further than the sample.

A strong answer does not start with a list of interface fixes. It starts by separating the objects. The design model is what the pump's engineering team believes about rate setting; the system image is what a nurse meets at the bedside, including the keypad, the displayed units, the confirmation step and the training card; the nurses' models are what they built from the old pump and from the new image. Norman's claim is that the nurses' models are built by interpreting the image, not by receiving the design model, so a plan that begins with retraining has skipped the mechanism. The two gulfs then organise the diagnosis. A rate set ten times too high is a gulf of execution failure if the keypad makes the intended rate hard to express, for example if the decimal entry requires a mode the nurse must remember; it is a gulf of evaluation failure if the pump does not make the entered rate and units unmistakable after entry, so the error is only discovered later. The two gulfs also say where the evidence lies: the execution side is tested by watching what the controls allow, and the evaluation side by checking what the display communicates after the action. Kempton's finding matters because the nurses' models are probably not absent but wrong in a workable way. A valve-style model of the old pump, in which turning the control up raises the rate faster, may have guided thousands of correct actions, and it would be reasonable to expect the same model to transfer to the new device. That makes the failure a prediction to test rather than an assumption: interview and observe nurses handling both pumps and record the rules they state and the actions they take, so the model is specified rather than inferred from an incident. The measurement part of the plan is where the reviews bite. Staggers and Norcio report that studies of this kind often fail to describe the method, leave the unit of analysis open, and record an interaction effect of design and user as a user defect, so the plan must fix the unit of analysis in advance. State that the object is the nurse's account of the pump's mechanism plus the actions in the log, define the coding scheme before collection, report agreement between two coders, and report the elicitation method in enough detail for another team to repeat it. Card, Moran and Newell show what the payoff of a complete specification looks like: their claim is checkable because the task, the device, the operator set and the selection rule were named, and the prediction of method choice was confirmed at about 80%. The equivalent here is a stated prediction, made before the redesign is tested, that a specific misreading of the rate display will occur or will not occur. Finally, the answer must say what would count against the plan. If a comparison of the old and new pumps shows the same error rate after the display is changed and the keypad is not, then the display was not the problem; if nurses state a correct model of the rate control and still set the wrong rate, then the model is not what governs the action and the plan needs to look at the physical layout of the task instead.

A strong answer begins by treating the task model as a plan, and the ward as the setting the plan cannot see. The designers assumed a sequence, scan, check, record, move on, and built the system to record the sequence. Suchman's finding is that nurses will read each step against what is happening around them, so the order of scan and check is negotiated with the patient, the colleague calling for help and the drug that is already in hand. The recorded times lag because recording is fitted into the work rather than driving it. Vera and Simon's reply matters here as a caution: the team should not conclude from this that no model is possible, only that the model was written for a situation that never occurs. Lave's evidence that the same people use different methods for equivalent problems in different settings predicts that competence on paper charts will not transfer to the screen, so a period of apparent incompetence is expected rather than a training failure. Hutchins supplies the unit of analysis. The relevant memory is not in the nurse's head but across the trolley, the wristband, the order and the display, so the question is what the joint system remembers and where a step is dropped. Schmidt and Bannon name what the task model omitted: the articulation work of arranging who checks what with whom, which is invisible in the model and continuous in practice. The study should therefore observe whole medication rounds, record the sequence of actions and the interruptions that reshape it, and count near-misses against the points where the sequence was renegotiated rather than against individual nurses. Anderson sets the boundary on what this yields: observational findings inform the representational scheme the system imposes, they do not supply a specification, so the design question becomes which representation supports the work as it is actually organised. The diagnosis would be wrong if nurses on wards with the same interface and lower interruption rates showed no lag, which would point to workload rather than to the model, or if the lag disappeared once the display changed without any change to the surrounding work.

The proposal mixes paradigms without saying so, and the first task is to separate them. Task completion times, error counts and a satisfaction questionnaire belong to the cognitivist reading: they assume a user pursuing a goal, and they treat a slow or failed task as the failure to be explained. That standard cannot see a clinician who completed the task correctly while arranging, with a colleague, who would check the dose, because the arrangement is outside the model. The observation belongs to the situated reading, and it is the part of the design that could see what the measures exclude: work that is renegotiated around the screen, the order and the patient, and the points where recording is fitted into the ward rather than driving it. Harrison, Tatar and Sengers give the placement its vocabulary, and their warning applies here, that components of a study take their meaning from the paradigm they sit in, so a timing figure and an observational note do not add up to one result. Sellen and Harper show what the mixed method has to watch for, since the paperless office failed because the ways people worked did not change, not because the technology was worse. Bodker's waves explain why the team reached for both sets of measures, since a hospital ward is a second-wave setting studied with third-wave ambitions about experience. Rogers supplies the reason for the recording I recommend: imported theories have broadened the field's discourse without much changing practice, so the team should state in advance which paradigm each measure serves, what result would count as failure for it, and what the observation would have to show for the cognitivist reading to be vindicated. My placement would be wrong if the timings and the error counts predicted the near-misses the observation finds, since that would show the goal-based model seeing the failure without help.

The brief makes three kinds of claim and the literature treats them very differently. The speed claim is the best supported, since Shneiderman (1983) reports that display editors took half the overall performance time of line-oriented editors and Nielsen (1993) gives 0.1 second as the limit for feeling that one is directly manipulating objects, but Nielsen also shows those figures were packaged from work done thirty years earlier rather than tested for the chapter. The learning claim, that the interface will be easier to learn, rests on Shneiderman's assertion that novices learn basic functionality quickly, not on a trial, and Shneiderman's own problems section warns that icons can take as long to learn as words. The error claim, that the interface will be safer, is the one the trust most needs and the least supported: Shneiderman says users feel less anxiety because actions are reversible, Hutchins, Hollan and Norman (1985) explain the feeling of directness through engagement and distance without measuring error, and Raskin (2000) argues from habit that modes, in which the same action produces different results, are a primary avoidable source of error, which is exactly what a folder-drag can create. Frohlich (1997) reports the supporting tests as a small number showing disadvantages and an even smaller number showing naturalistic choice of alternatives. To separate the claims, the trust should compare a reversible drag version with an irreversible one at the same response latency, and count unrecovered filing errors and second-attempt times, especially for clinicians returning after a gap; if matching latency erases the difference, the speed account is sufficient. The strongest case for delegation is Gentner and Nielsen's (1996) argument that with many objects direct manipulation becomes repetitive drudgery, reinforced by Maes (1994) on agents that take over routine work; Horvitz (1999) supplies the way to decide, by comparing the expected cost of a wrong automatic filing against the cost of asking, and Shneiderman and Maes (1997) show the two positions disagree about who initiates rather than about predictability. Evidence would decide it: if mis-filings would harm patients, the expected cost favours asking; if the gesture is rare and cheap to reverse, manipulation wins.

The claim makes two statements and supports neither as written. Against ISO 9241-11:2018, usability belongs to a use, so the score is evidence about the users, goals and context that were tested, not a property of the app. Nielsen (1993) separates learnability, efficiency, memorability, errors and satisfaction, and the scale measures a subjective rating, not effectiveness or efficiency. The number is also weaker than it looks. Brooke (2013) shows that the System Usability Scale was developed in 1986, has never been through a formal standardisation process, and carries a nought to one hundred range that its author calls a marketing decision, so the score is a perception summary of unknown provenance for this product. Hornbaek (2006) found that 57 per cent of 180 studies measured task completion time and only 13 per cent measured task completion time, accuracy and satisfaction together, and Hornbaek and Law (2007) found effectiveness and preference correlate at .164, so a preference score cannot stand in for effectiveness. The experience sentence is unsupported altogether, because ISO 9241-210:2019 puts perceptions and responses from use or anticipated use, before, during and after use, inside the concept, and a single post-session score samples that poorly; Hassenzahl (2004) found perceived usability and beauty moved differently, beauty tracking identification while perceived usability tracked the problems encountered. A defensible rewrite names the population, the tasks, the setting and the measure, for example that among 20 first-time users booking a return trip on a phone, the median score was 84, while completion time and error rates were recorded separately. To claim experience, the team would need published items with reported reliability, measured before, during and after use, including anticipated use, following Bargas-Avila and Hornbaek (2011) and Vermeeren and colleagues (2010), and convergent evidence from a second instrument built on a different theory. The wider term would earn nothing if two such instruments agreed with the pragmatic measures in every case, which is the test Hornbaek and Law (2007) applied to usability and found wanting.

The document names the wrong variable. It treats automation as a single property and promises a reduction in workload, but the level and the type of automation are what decide the effect. Against Fitts (1951), the procurement is running the old comparison: it assumes that because the machine reads and flags data well, the nurse's share of the work shrinks with its error rate. Dekker and Woods (2002) show why the comparison cannot settle the design, since the outcome depends on how the nurse and the system coordinate once both are on the ward, not on which of the two is stronger at reading a number. What has been relocated is the interpretative work, and Bainbridge (1983) shows that the parts left to the operator are the parts that defeated automation: rare, ambiguous and high consequence. The nurse now monitors a system that is usually right, and Parasuraman and Manzey (2010) report that complacency appears under multiple-task load and cannot be removed by practice, while Endsley and Kiris (1995) found that monitors were slower to take over after a failure than people who had done the task manually. The workload promise is therefore unproven and might be false. The design decisions that matter are the level (Sheridan and Verplank, 1978), the type, since data acquisition, analysis and action selection can sit at different levels (Parasuraman, Sheridan and Wickens, 2000), the reliability and error structure of the aid, because Wickens and Dixon (2007) place the crossover at about 0.70 and show that misses and false alarms cost differently, and the visibility of the system's state, since Norman (1983) distinguishes slips, caught by feedback, from mistakes, corrected only by a better model. Kaber and Endsley (2004) show the level is a real lever, because intermediate automation raised situation awareness in their experiment. The ward should measure detection performance by the nurse unaided, time to take over after a correct and an incorrect flag, the rate of missed detections and false alarms, and trust before and after the change, since Lee and See (2004) and Hoff and Bashir (2015) make calibrated trust the aim rather than more trust. A result that would count against the case is slower and less accurate take-over than the unit's previous manual practice, or a rise in missed deteriorations, because that would show the relocation cost more than it saved.

The report answers the wrong question. It shows that twelve users drawn from a general panel could complete the form, and a completion rate of above ninety per cent establishes that the form is usable by people like those participants; it does not establish that the form is usable by the specified users the benefit is for. The standard's own logic supplies the objection, because IEC 62366-1 frames usability as a process in relation to safety and the FDA guidance asks for evidence about the intended users, uses and use environments, so a sample that omits low-literacy claimants, claimants on a shared metered phone and claimants reading in a second language leaves the intended-use claim untested. Reading the omission as an individual support matter repeats the fallacy Karsh and colleagues identify: the technology does not improve access by itself, it shifts the work, and here the work moves to the claimant in the form of interpretation, connectivity and a device they may not control. Reason's model locates the cause in latent conditions and in the design that shaped the act, so a claimant who abandons the form has not failed; the form failed a person it was built for. The agency has also mis-specified the system. Sittig and Singh place the human-computer interface in an eight-dimensional model whose dimensions interact, so a form redesigned while the literacy demand, the device assumption and the support route stay as they are is a partial repair. Rajkomar and Blandford show why this matters in a care setting, since cognition and safety were distributed across people, layout and artefacts and a device-level test recorded none of it. The exclusion the disability organisations name is also social, not only technical: Shinohara and Wobbrock found that devices mark their users and that a functional need outweighs the embarrassment of that mark, while Bennett, Brady and Branham show that the independent individual user assumed by the design hides the support networks that make access work. What the agency should do follows from these findings. It should recruit participants who meet the specified-user description, including the three groups named, and it should observe the form where those claimants actually use it, on their own devices and connections, rather than in a lab. It should record abandonment and recovery, not only completion, because the harm of exclusion appears as a person who leaves. It should test the redesigned form with the users as decision-makers rather than as test subjects, since Bennett and Rosner show that a designer's empathy is not the same as the user's account, and Frauenberger and colleagues give the criterion for judging the process: the judgements must be inspectable and the account owed to the participants, not only to the agency. A result that would count against the service is a redesigned form whose specified-user participants complete it at a rate comparable to the panel and abandon it at a comparable rate, with the claimants themselves recording that they could complete it without help. If, after that, claimants still needed assistance the form was supposed to remove, the agency's claim would not survive.

The evidence pack establishes that, in one setting and one period, agents with the assistant closed more tickets per hour than a comparison group, and that the agents who replied rated the assistant highly. Neither figure supports the claim that the paradigm is validated. Noy and Zhang's preregistered experiment with four hundred and fifty-three professionals is the standard the pack would have to meet, because it randomises access, uses a task with stakes and reports both time and rated quality; the pack reports one output and no randomisation, so it cannot separate the assistant from the queue mix, the staffing or the season. Dell'Acqua and colleagues then warn that the effect has a boundary, since consultants did better inside the model's capability and worse than controls just outside it, so a claim about all tickets is an over-reach until the pack shows where its own boundary lies. The second claim, that the assistant improves judgement because agents review the draft, is not an empirical finding at all. It is a design proposal of the kind Weisz and colleagues present as principles and Amershi and colleagues present as guidelines, and it should be judged first by whether it helps a designer decide and second by whether its prediction holds. The audit of Cockburn and colleagues applies to the pack directly, since a single comparison with no stated power, no registration and no negative result is exactly the practice their review finds unreliable. Read through Wobbrock and Kientz, the pack mixes a contribution type: the interface reasoning is artefactual and the 'more tickets' sentence is empirical, and each needs its own standard. The process evidence says what to add. Wadinambiarachchi and colleagues found fewer ideas and lower originality with generative support, Barke and colleagues found verification and redirection absorbing much of the saved time, and Lee and colleagues found tool confidence predicting less self-reported critical thinking. A pack that measures throughput alone cannot show whether review is real or nominal. The evidence the team should gather is therefore a preregistered comparison with adequate power, agents assigned at random, throughput and reply quality rated blind, the proportion of drafts edited, the time spent verifying, and a measure of whether agents can detect an incorrect draft when one is planted. The result that would count against the team is straightforward. If trained agents fail to catch planted errors more often than agents without the assistant, or if blind-rated reply quality does not rise, then the review step is not delivering the judgement the team claims, whatever the ticket rate shows. On the wider dispute, Shneiderman would ask whether the controls keep the agent in charge; Bender and colleagues would ask who bears the cost of the model, a question no internal usability study can answer.

The fault is duplicated state. Two values are supposed to describe the same fact. Nothing forces them to stay in agreement, and every code path that changes one is a chance to forget the other. The number of unfinished tasks is not independent information. It is derived from the list. The corrected design keeps the list as the single source of truth. It computes the count when the count is displayed, or immediately before rendering. The two displayed values then cannot disagree, because they come from one place. Write the tests against the logic, not against the pixels. First, a unit test that adds three tasks, completes one and then asserts that the derived count is two. Second, a test that renders from a given state and asserts that the output depends only on that state. A change to the state then produces the expected output and nothing else. The important point is that the fix removes the class of bug. A check that the two numbers agree would only detect it, because both copies would still be there.

The evidence does not support the rule as stated, though it does not forbid the practice either. Two studies point in different directions. Fucci and colleagues analysed 82 development tasks from 39 professionals and found that the order in which tests and production code are written had no important influence; what was associated with better outcomes was granularity and uniformity, meaning small steps taken steadily. Against that, Nagappan and colleagues studied four industrial teams that had adopted test-first, and reported pre-release defect density down by between 40 and 90 per cent, at a cost of 15 to 35 per cent more initial development time. Neither settles the question. The Fucci study is small, set in industry, and measures quality only as functional correctness. The Nagappan study is four case studies of teams that chose the practice, with no control group, so their own discipline may explain the gain. A blog post cannot outweigh either, and the case-study figure does not prove causation. My recommendation is to require tests for all behaviour, together with small, steady steps, and to leave the order of writing to the developer. That keeps what both studies associate with better outcomes and drops the part that is disputed. To check the decision, I would measure the defect rate found after delivery and the time a change takes from first commit to merge, comparing a pilot team with a similar team rather than trusting a single before-and-after number.

A strong answer designs the study rather than restating the argument, and it begins from the two accounts the unit names. That means two matched cohorts, taught the same language and the same domain context, with one group receiving explicit notional-machine instruction and the other not, so that the only difference is whether the model of execution is taught deliberately. The measures must separate the two theories. To test the model account, measure tracing and state prediction on unfamiliar code, using the kind of task Lister and colleagues used, because that task is the closest available proxy for the model itself; add a writing task so that a tracing advantage that never reaches writing is visible as such. To test the context account, measure engagement, persistence and self-reported interest, and include a third condition in which the context varies while the machine instruction is held constant, which isolates context from instruction. Then state what would count against the thesis, because that is where the design earns its keep. If the notional-machine group traced unfamiliar code better but the context group wrote equally good programs and persisted longer, the thesis would be weakened, since the model would be shown to be neither necessary nor sufficient for the outcome that matters. If, instead, the model group outperformed on tracing and that advantage carried into writing, while context alone moved only engagement, the thesis would be upheld. A first-class answer also names a limitation it cannot remove, such as the difficulty of measuring a mental model at all, and says how the study would handle learners who arrive with a model already, as Pane and colleagues found.

A strong answer designs a comparison and names what each result would mean. The unit's thesis is that the deficit is a model of the machine, stable across languages, so the first move is to hold the task and the operations constant and vary only the representation. Give one matched group the same element-removal task in the immutable language, where assignment cannot create an alias, and another group the task in the shared-reference language, and measure both whether the students predict the caller's list correctly and whether they can explain why. The model account predicts that the surprise survives the change of representation, because the student still lacks a picture of what a name refers to; the rival account, after Fisler (2014), predicts that the difficulty appears only where the operations allow it. The measures should separate the two accounts rather than lean on one outcome. Prediction on unfamiliar code, of the kind Vainio and Sajaniemi (2007) used, stands for the model, and a transfer task in a third representation tests whether a taught model travels, since a model that is real should survive the move. The result that would count against the thesis is a group that, taught the shared-reference language alone, never shows the error once the representation changes, because the error would then belong to the language rather than to the student's model. A first-class answer also notes the hard part, that a mental model is not directly observable, so prediction and explanation are proxies rather than the model itself, and it checks, after Sirkia and Sorva (2012), whether the student could state the state the program was in at each step.

The freeze follows directly from run-to-completion on a single thread. The event loop drains one task at a time, so while the click handler is running, every click and keystroke is queued and unhandled (WHATWG, 2026; Mozilla Foundation, 2026). Three seconds exceeds the roughly one-second limit at which a person's flow of thought is interrupted and approaches the ten-second limit for holding attention, so the delay is not merely slow but changes the interaction (Nielsen, 1993; Miller, 1968). The fix is to stop holding the thread. Start the request, return from the handler immediately, and let the reply arrive as a continuation: a promise, or an awaited step inside an async function, chains the work that builds the table without blocking (Haverbeke, 2024). If the reply needs heavy processing, move that part to a worker so the thread that answers the user stays free. To show the fix works, measure the time from click to the interface regaining responsiveness, not the total time to the table: the work is no faster, so the table still takes three seconds, but the cancel button and scrolling answer at once. State plainly what it does not fix: the request is still slow, and the handler must now handle the failure case, because a promise that rejects needs a state the earlier code did not have. This mirrors the misuse Okur and colleagues (2014) found, where asynchronous constructs are used without attention to the states they introduce, and it is the kind of fault Turcotte and colleagues (2022) detect in widely used code.

The screen-reader user receives not the drawn control but the accessibility tree the browser derives from the DOM, which carries a name, description, role and state for each element (Mozilla Foundation, 2026). A pair of div elements has no native role, so the derived tree will not announce a switch at all; the state held in a script variable and shown by colour has no counterpart in the tree, so even if a role were supplied the on and off states would not be conveyed unless an explicit state property such as aria-checked is set and kept in step (W3C, 2023b). Two ways the control is therefore present in the markup but absent from the interface are missing role, so the user does not know a switch is there, and missing state, so the user cannot tell whether it is on. This is the same missing-semantics failure Ross and colleagues found at scale in native applications, where 92.0 per cent of floating action buttons lacked a usable label (Ross et al., 2018). To fix it, give the control a real role and state, keep the accessible name short and descriptive, ensure keyboard operation and visible focus, and, where practical, prefer a native control whose semantics the browser derives for free rather than teaching the tree by hand. To show the fix works, measure what a screen-reader user encounters, listening to what is announced on focus and on change, or run automated checks as a floor; do not measure the number of criteria passed, because Power and colleagues found only 50.4 per cent of real user problems were covered by the guidelines and some persisted even where techniques were implemented (Power et al., 2012). What the fix does not do is guarantee that the announcement is understandable in a user's context of use, since accessibility, on Petrie and colleagues' account, is about using the site across contexts, and Vigo and Harper showed users still develop coping tactics when a page creates problems (Petrie et al., 2015; Vigo and Harper, 2013).

The bug is the class that separation is meant to remove: the same fact exists in more than one place and the copies have drifted. Fowler describes a form as holding record, session and screen state, and warns that keeping screen state and session state synchronised is an important task that data binding only partly solves because change can cycle between a control and the underlying value (Fowler, 2006). Here the range lives in a script variable, in the input fields and in a label, so a repaint that reads the fields can disagree with the variable the query uses. I would introduce a single source of truth for the range, a plain object holding the start and end dates, and treat the fields and the label as views that are written from it rather than read from. That is the original division: Reenskaug's models, views and controllers, with each view associated with its model (Reenskaug, 1979), and Krasner and Pope's three-way factoring that gives the application domain, the display and the user interaction to different objects (Krasner and Pope, 1988). Parnas gives the reason the seam should fall there rather than elsewhere: hide the decision likely to change, and how a range is displayed is exactly that decision (Parnas, 1972). The separation is not free. It creates new code that formats the range for the label and maps it into the fields, and, if I add a computed value such as the number of days, new derivation code, and this is the logic nothing tests until a test is written, the thin wedge Feathers concedes remains at the interface (Feathers, 2002). If I push further to passive view, the widgets become entirely humble but I must build a stand-in for the screen during tests, extra machinery that has to be kept truthful (Fowler, 2006). I would test the range object and the formatting directly, with no widgets present, by setting a range and asserting what the view was asked to display, following Feathers' method of driving the smart object against a stand-in view (Feathers, 2002). What remains hard to test is that the stand-in still matches the real screen, since a double that drifts from the widgets is itself a new pair of copies that can disagree. If the team later adopted a reactive state layer or the proposed signals primitive, the manual propagation would be automated, but correctness would still depend on glitch avoidance, a named open problem in the reactive survey, so an inconsistent derived value remains possible even then (Bainomugisha et al., 2013; TC39, 2024).

The code collapses at least three distinct failures into one string, so the interface cannot act on any of them. The first is unreachability, which the Fetch Standard defines as a network error: a response whose type is error and whose status is 0, with an empty header list and body (WHATWG, 2026). That is the tunnel case, where the request never became a conversation, and the remedy is a retry the person controls, because the program cannot know when the network returns. The second is refusal, a server that answered with an error status. RFC 9110 makes the status informative: a 503 Service Unavailable may carry a Retry-After that tells the client how long to wait, while a 501 or a 403 will not change by waiting (Fielding, Nottingham and Reschke, 2022), so the design should read the code rather than render a generic message. The third is slowness, the state that has not yet resolved; without a timeout the client cannot separate a slow answer from no answer at all, which is the hang, and Dean and Barroso show why the tail is the design case rather than the average, since a request waiting on many parts is slow as often as its slowest part (Dean and Barroso, 2013). Nygard supplies the vocabulary, a timeout on every remote call, a circuit breaker for a dependency that keeps failing, and a bulkhead so one failure cannot drain the rest (Nygard, 2018). The empty list is the fourth case the code never modelled, a successful response with no results, and it belongs in the interface because a person reads it differently from an error. The contested position is local-first software: Kleppmann and colleagues argue the primary copy of the data should live on the device, with synchronisation in the background, so the network becomes optional and unreachability need not reach the interface at all (Kleppmann, Wiggins, van Hardenberg and McGranaghan, 2019). It does not apply fully here because the restaurant data belongs to a third party and is computed on a remote machine, so the client cannot hold an authoritative copy; for facts you do not own, the failure states are the only design available. What remains is to represent them, so that a blank list, an offline message and a refusal are visibly different to the person.

The student has a comprehension problem rather than a tooling problem: the breakpoint shows them the state where they looked, and the fault lies earlier, in the path that applied the discount, so what they lack is a correct model of what the program did rather than a view of a value. The tool that would help is one that answers a question about the run, in the way Ko and Myers designed the Whyline, which let programmers ask why an output occurred or why it did not and answered with the relevant runtime events, cutting debugging time by nearly a factor of eight on identical scenarios (Ko and Myers, 2004). A fault-localisation tool such as Tarantula would narrow the search rather than explain the run: it ranks statements by how strongly their execution tracks failing rather than passing tests, and it beat four comparable techniques on the same subjects (Jones and Harrold, 2005), but a ranking says only that a statement is suspicious, and Parnin and Orso showed that the assumption a developer can understand and repair a bug from the faulty statement alone does not hold, because a root cause usually requires navigating dependencies and running the program with different inputs (Parnin and Orso, 2011). The minimal experiment is the student's own: take two inputs that differ only in the discount code, run both, and compare the value of the discount at each step where it is computed, which is a hypothesis about one variable tested by observation rather than a walk through the whole function. That is the procedure Zeller sets out, reducing the failing case until what remains is minimal and then following cause and effect through the run (Zeller, 2009). The student should also record what they expected the value to be at each step, because the faults that Eisenstadt's informants called hardest were the ones that broke an assumption, and a wrong expectation is only visible when it is written down (Eisenstadt, 1997).

The lead has taken a single task and drawn a general conclusion, and the controlled evidence separates two things the rule has merged. Fucci and colleagues, with 82 tasks from 39 professionals, found that the order in which tests and code are written had no important influence, while fine-grained, uniform steps were positively associated with quality and productivity (Fucci et al., 2017). Tosun and colleagues found no statistical difference in quality between test-first and incremental test-last, a productivity gain for test-first on a simple task and a significant drop on a complex brownfield task (Tosun et al., 2017). The legacy change is that brownfield task: the code resists small steps, so the tests written first keep changing and the cost lands on the developer. What the team should keep is tests throughout and small steps; what they should drop is the badge of writing every test before any code. Santos and colleagues' family of twelve experiments found the task rather than the order determined quality (Santos et al., 2021), and their design, replicating across tasks, is the measurement the lead should copy: hold the task type, the developer and the step size constant and compare the two orders across several features.

The lead has judged both policies against a single measure, defect escapes, that the evidence does not identify as their main product. Bacchelli and Bird observed 17 developers across 16 teams at Microsoft, classified 570 review comments, and surveyed 165 managers and 873 programmers: finding defects was the top motivation for 44 per cent of the programmers, but defect comments were about 14 per cent of those classified, well behind code improvements at 29 per cent, while the outcome the study observed was knowledge transfer, team awareness and shared understanding of the change (Bacchelli and Bird, 2013). Review may be paying for understanding that reduces cost later, which a defect count will not show. Gousios and colleagues studied almost two million pull requests and 291 projects and found that around 84 per cent were eventually merged and 60 per cent processed within a day, but that review did not raise the chance of a merge and did slow processing, with 53 per cent of rejections for distributed-process reasons rather than the code (Gousios et al., 2014). That is the slowdown the lead is feeling, and it is the price of coordination rather than evidence that review failed. On co-location, Herbsleb and Mockus found distributed work items took about two and a half times as long as collocated ones because more people were involved (Herbsleb and Mockus, 2003), while Bird and colleagues found a negligible quality difference between distributed and collocated components, which shrank further once the number of developers was controlled (Bird et al., 2009). Co-location should therefore be judged on time and coordination, not on defect escapes, and a merge tool cannot be judged on defects either: Perry, Siy and Votta found that only 3 per cent of changes that went to the same file physically overlapped, so most interference is semantic and invisible to the merge (Perry et al., 2001). To decide, the team should measure what review is meant to buy, such as the time new members need to make a safe change, and compare matched changes under the two working arrangements rather than the whole product, the design Bird and colleagues used for location (Bird et al., 2009). Nelson and colleagues' life-cycle model also suggests watching how often conflicts are deferred, since 56.18 per cent of their participants had deferred at least one and deferral moves work onto the team (Nelson et al., 2019).

The two claims are different in kind, and the engineer has merged them. The Fitts's law comparison predicts movement time from target distance and width, but Soukoreff and MacKenzie (2004) showed that the comparison is only meaningful under declared conditions: with four formulas in use (Drewes, 2010) and studies inconsistent enough to obstruct comparison (MacKenzie, 1992), the index formulation, its range and the adjustment for accuracy must be stated before a lower index of difficulty can be read as a faster target. The keystroke-level result is a different sort of claim. Card, Moran and Newell (1980) predicted task time on ten systems with 21 per cent error for individual tasks, but their mental operator was estimated from the same data the model was tested on, and the validation excluded task instances where the user did not follow the prescribed method. The prediction that the palette method is fastest therefore assumes a user who adopts the prescribed method and is accurate to about a fifth of a task time, which is not the same as showing that the palette is the method people will use. Gray and Fu (2001) show that people adopt least-effort tradeoffs even when the effort difference is small and even at a higher error rate, so a faster method may simply be ignored. The decision support that would settle it is a within-subject comparison of the two designs with the same commands, declaring the Fitts's law formulation and index range, reporting movement time and end-point scatter, and measuring both the time taken and the method chosen under each design, so that the timing claim and the choice claim can fail independently.

For each command the index of difficulty is a logarithm of distance over width, ID = log2(D/W + 1) in the formulation MacKenzie recommends, and log2(2D/W) in Fitts's own; either way a radial menu shortens the distance the pointer travels and can enlarge the effective width, because the inner ring gives more usable space than a toolbar slot. The model therefore predicts a lower index of difficulty and a shorter movement time, and it predicts diminishing returns, because the gain is logarithmic: halving the distance is worth about one bit, not a halving of time. Rely on the model for that relative ranking, where the fit is reliable, and do not use it for an absolute prediction such as a target time in milliseconds, because its information-theoretic basis is disputed and a speed-accuracy trade-off account is an alternative. It says nothing about learning, search or error once the menu is radial, nothing about visual crowding on the outer ring, and nothing about rare but destructive commands that should not sit beside frequently used ones. Those claims need their own evidence. Run a within-subjects comparison of the two layouts with the same commands for every participant, counterbalance the order, and report mean movement time with a spread and a sample size. Include participants of different ages and at least one who does not use a pointer, and report their times beside the pooled mean. The standard error is the standard deviation divided by the square root of the number of observations, so twenty participants give a precision of about 0.22 times the spread and eighty about 0.11 times; halving the error costs four times the data. A difference smaller than that precision is not evidence of improvement, and the honest claim is conditional on the tasks and the participants tested.

Model each structure as a graph with settings as vertices and navigable moves as edges. The three-level tree has exactly one path to each setting, so its depth is the number of steps from the root, and the counterexample to the claim that everything is easy to find is any setting at the third level whose category name the user does not guess. Depth is a graph quantity: the maximum over settings of the shortest path from the root, which breadth-first search computes directly. Depth is also not findability, because a graph says nothing about whether a label is understood. Flattening the structure into a list removes that depth but adds a search step that depends on the query the user types, so the state becomes the query and its history rather than the screen, and the graph is no longer a tree. A halving search over ninety settings needs at most seven comparisons, because 2 to the seventh power is 128, but that bound assumes the list is ordered and the query is known. Flattening also does not remove the counting problem: ninety settings with several values each produce a large state space that the test plan must cover. If the settings are independent, exhaustive coverage is the product of the option counts; a pairwise plan covers each pair of settings in at least one run and is far smaller, but it cannot show the absence of a fault that needs three settings at once. Those detection figures come from a single report and its own studies, so treat them as a reason to prefer higher strength, not as a guarantee. Reachability is the other guarantee to test: one focusable control that no path reaches is a single counterexample, found by search rather than by sampling users.

A strong answer names one interface and one claim, then writes the claim in the form the unit demands before judging anything. 'Every action can be undone' becomes: for every reachable state, there is a sequence of inputs that returns the user to the state before the action. What refutes it is one reachable state with no returning sequence, for example a modal dialog that traps the user until a field is completed. What would establish it is a check over every reachable state, which is expensive, so the claim is cheap to falsify and costly to prove. The answer should then choose. If a false claim would be trivial, as it usually is for a settings screen, a written invariant is the right level: the claim is stated precisely and reviewed, no tool enforces it, and that is proportionate. If a false claim would be fatal, as it was on the Therac-25, the argument of Bowen and Hinchey and the industrial record of Woodcock and colleagues both point the other way. The formal search is worth its cost where regulation or consequence forces the issue, which is why Campos and Harrison compile an interface model and search it for a violating trace, and why Bolton and Bass put the operator's procedure inside the model. The answer should end by naming what would change the judgement, such as evidence that the model's counterexample matched a failure in the field, which is the test the unit takes from Fetzer's objection.

A strong answer fixes the comparison before measuring. It takes one command set, such as a drawing tool with a pointer mode and a pen mode, and implements it in at least three versions that differ only in how the state is handled: a system-maintained mode with a visual indicator, which is the unit's prescription; a system-maintained mode with kinesthetic feedback, following Sellen, Kurtenbach and Buxton's foot pedal; and a user-maintained quasimode that lasts only while an action is held, following Raskin. If the command set permits it, a modeless version that removes the distinction should be the fourth arm. Participants should be tested at two levels of expertise and again after a delay, because the prediction that matters is that training will erase the differences between the versions, leaving the removal account no advantage. The measures should be the error rate, defined as an action taken in the wrong state, the cost of recovering from such an error, and time on task, since a version that reduces errors by slowing everyone down has not settled the question. The answer should say what would settle it. If the quasimode version reduces errors and does not raise time on task, then Raskin's programme is better supported than the unit's prescription and the unit should say so, citing Raskin against Sellen. If the signalling version matches or beats removal once expertise is controlled, the prescription survives, and the unit can cite Degani and Heymann's demand that the display track the machine state as the reason. The answer should also note what the study cannot settle, which is whether the states the design contains are necessary at all, and it should say what would settle it.

A strong answer separates the two quantities the unit distinguishes. The proposal changes the topology, from three levels to one, and the evidence says topology alone does not determine the cost. Snowberry, Parkinson and Sisson found that search improved as menus grew broader across four 64-item hierarchies, which supports flattening, but their randomly arranged version of the single broad menu fell outside that trend, so breadth without a usable labelling scheme is not the same intervention. Kiger's recommendation of eight or nine items per level points the other way, since a single page of 64 controls is far beyond it, and the answer should say that the two studies disagree rather than choosing one silently. Burnett, Lawson, Donkor and Kuriyagawa give the resolution the unit uses: ordered menus gave a logarithmic cost in breadth and favoured the shallowest, broadest structure they tested, 16 by 3, while randomly arranged menus gave a quadratic cost and favoured compromise structures of 4 by 6 or 8 by 4. The decisive question for the product team is therefore whether the 64 control labels can be ordered and grouped so that the flat page works as a structured menu, and whether the control's label predicts its content well enough to be found. Pirolli and Card and Fu and Pirolli supply the mechanism for judging that, since the click is predicted from the link text, so the answer should propose scoring the labels before the redesign rather than after it. The measurement plan should compare the path taken with the shortest path in each structure, since a flat page guarantees a short path by construction and the interesting quantity is whether people find it, together with time on task, selection errors and the rate at which users abandon. It should use more than one task type, because Burnett's result is that the best structure depends on whether the cues can be anticipated. The result that would show the proposal to be wrong is a flat page on which time to the three-deep control rises and abandonments increase while the ordered, grouped two-level alternative does better, which would mean the team had optimised a graph distance and left the labels unchanged. The answer should also note that Aula, Jhaveri and Kaki found experienced users rely on their own structures and still struggle to re-access information, so a before-and-after comparison should include re-finding, not only first search.

A good answer starts from the two quantities the unit separates. The 4,096 figure is correct as a count of the state space by the product rule, and it is computable from the model before any code exists, which is the first move in the category-partition method that Ostrand and Balcer (1988) describe. It is not the size of the plan. Cohen, Dalal, Fredman and Patton (1997) report that a covering array can cover all t-way combinations with a test set that grows only logarithmically in the number of parameters, and the NIST tutorial makes the gap concrete by covering ten binary settings for all three-way combinations in thirteen tests against 1,024 configurations (Kuhn, Kacker and Lei, 2010). The answer should therefore replace the random sample with a covering array at a stated strength, chosen by deciding which interactions the design could plausibly get wrong, because the strength is a chosen t and not a property of the system (Hartman and Raskin, 2004). The guarantee is exact and narrow: every combination of the chosen number of switches is exercised, and nothing is claimed about a fault that needs more switches at once, so the plan cannot show that no fault remains. The answer should also state the two assumptions the equivalence needs, no dependence on complex event sequences and a small set of discrete values (Kuhn, Wallace and Gallo, 2004), and note that twelve binary switches satisfy the second but not the first, since a settings panel is applied in sequence and mode can matter. The measurement plan should record which combinations the array covered, and then, following Yilmaz, Cohen and Porter (2006), use any failure it finds to characterise the subspace that causes it rather than only to report a failure. A result that would show the plan inadequate is a fault whose smallest triggering set needs more switches than the chosen strength; a second is a fault that appears only for a particular order of settings changes, which the array, being orderless, cannot reach. The answer should also say that the random sample of 200 has no stated coverage, so it cannot be defended either way, whereas the array's coverage can be computed and checked.

The team has multiplied three success probabilities, so it has used the multiplication rule, which Feller (1968) licenses only for independent events. If the three step failures were instead treated as disjoint, the probabilities of failure would add rather than multiply, and the unit's two-step case shows how far apart the two answers are: 0.04 by addition against 0.0396 under independence, a difference equal to the chance that both steps fail, which the sum counts twice (Grinstead and Snell, 2003). Neither rule is more accurate as arithmetic; the rules differ in what they assume, and the report does not say which assumption it made. The three component figures are human error probabilities of the kind tabulated by Swain and Guttmann (1983) for probabilistic risk assessment, and French and colleagues (2011) argue that such numbers rest on assumptions that are rarely justified, including the decomposability of tasks and the independence of failure modes, so the product inherits the uncertainty of three estimates and adds an assumption. Swets (1973) removes a further constant from the calculation, since a step's error rate partly reflects where its decision criterion was placed, so it is a design choice rather than a measurement. The strongest case for treating dependence as a design variable is Littlewood and Miller (1989), who show that diverse methodologies decrease the probability of simultaneous failure and that better than independent behaviour is theoretically possible, with Eckhardt and Lee (1985) supplying the intensity function that makes the effect a parameter rather than a hope. The strongest case against the single figure is Knight and Leveson (1986), whose 27 versions failed together substantially more often than independence predicted, which is what makes the report's silence about dependence a defect rather than an omission. The reasoning literature widens the objection: Casscells and colleagues (1978) and Eddy (1982) show experts answering the wrong conditional, and Gigerenzer and Hoffrage (1995) show the answer improving when the same data are stated as frequencies, so the correction is partly a matter of how the estimate is written. A study that would settle it would pre-register a set of checkout steps, log joint failures across the whole flow rather than per step, estimate the dependence between steps rather than assume it, and report how far the flow-level figure moves across the plausible range of dependence, using frequency counts as Gigerenzer and Hoffrage recommend. If the joint failure count matches the product within sampling error, the team's figure survives; if steps fail together more often, the 94 per cent is an artefact of the arithmetic rather than a property of the flow.

The developer is using the law predictively, quoting an absolute time as a property of each layout, and that is the use the sources do not support. Fitts (1954) established the ordering this unit keeps: response duration grows with an index of difficulty built from the logarithm of the ratio of amplitude to target width, and Card, English and Burr (1978) showed the law can rank devices, with the mouse fastest and the lowest error rate and more than 80 per cent of movement time variation captured by the regression equations. That is the strongest case for ranking A and B, and it is a real one, because A has larger and closer targets on both terms of the index. The case against quoting a difference is threefold. First, the warrant is an approximation: MacKenzie (1989) showed that Fitts cited an approximation of Shannon's Theorem 17 that is useful only where the signal-to-noise ratio is large, and MacKenzie (1992) records that Fitts's own conditions ran the amplitude-to-width ratio as low as 1 to 11, so a layout of large close targets is exactly the region the derivation excluded. Second, the summary statistic is contested: Zhai (2004) argues that throughput conflates information with non-informational properties of device and task, while Soukoreff and MacKenzie (2004) specify how to compute it, so a single predicted time is a protocol figure rather than a measurement. Third, the fit is not a mechanism: Meyer and colleagues (1988) derive the logarithm from optimising movement time under noisy force production, Plamondon and Alimi (1997) argue that the many competing interpretations call the explanatory power of such models into question, and Balakrishnan (2004) shows that techniques which appear to beat the law do so by changing target distance or width. The touch literature shows how far the model still moves: Bi, Li and Zhai (2013) report better prediction from a touch-specific model at an R-squared of 0.91 or above, Woodward and colleagues (2020) find the standard law better for children at an R-squared of 0.93, and Ko and colleagues (2021) find the better width choice varies between datasets. What would settle the comparison is a within-subjects experiment on the real interface, reporting movement endpoints and error rates, fitting both nominal and effective widths and both fixed and free ambiguity terms, stating the index range and trial counts before the data are collected, and reporting throughput alongside speed and accuracy rather than instead of them. The result that would count against the developer is that the fitted difference between A and B falls inside the uncertainty of the fit, or that the ranking reverses when effective width and the error rate are included, while an observed difference that survives both checks and holds for older and younger users would support the ranking.

The proposal makes two claims of different standing. The first, that twelve equally likely options carry more uncertainty than a shorter set, is licensed: entropy is a property of the distribution, and equal probability gives log2 of the number of outcomes, so MacKay (2003) and Shannon (1948) support the comparison of two distributions over the same outcomes. The second, that the difference converts into seconds per decision, needs a rate, and the sources do not supply one. The strongest case for speed is Hick (1952), who found the rate of gain of information constant on average within one perceptual-motor act at about five bits per second across experiments with up to ten alternatives, and Hyman (1953), who showed that reaction time grows with the number of equally probable alternatives and depends on the information content of the stimulus; both are real results, and both point the same way as the proposal. The case against quoting a saving is fourfold. The rate is not one number: Longstreth and colleagues (1985) replicated a slope of about 15 milliseconds per bit against the usual 150 under conditions that excluded the standard explanations, and their replacement makes time a function of the number of alternatives. The relation is conditional: Proctor and Schneider (2018) list practice, stimulus-response compatibility, very large set sizes and sequential dependencies among the effects it does not handle. The block-level fit is not the whole model: Mordkoff (2017) shows that average uncertainty leaves trial-type differences inside the block unexplained, and that a second, trial-level predictor is needed, so a uniform set of twelve is a special case rather than a general price. And the time is composed of stages, not transmitted at a rate: Sternberg (1969) shows that additive factor effects index separate processing stages whose durations add, and Card, Moran and Newell (1980) price a unit task with analyst-chosen operators, predicting expert time to within about 20 per cent for error-free routine performance. What is missing from the proposal is what the user has to decide: Miller (1956) shows that recoding into chunks stretches the information bottleneck, so the entropy of the display is not the entropy of the decision, and Card (1982) treats menu selection as recognition rather than recall. The study that would settle it measures decision time for the twelve-option and shorter panels over enough trials to estimate both, with the usage distribution of the options recorded rather than assumed equal, with the arrangement held constant so that layout does not carry the effect, and with the analysis comparing a model in entropy, a model in the number of alternatives and a two-layer model. Tullis (1985) is the precedent for the arrangement control, since a broad and a deep hierarchy differed in errors at the same size. The result that counts against the team's figure is a fitted difference in time inside the uncertainty of the fits, or a slope that changes when the usage distribution is measured rather than assumed.

The two symptoms have different causes and call for different evidence. The mis-selection is a coordinate mismatch. The tap arrives in screen coordinates, and a hit test that runs in screen coordinates fails for a rotated or scaled object, because the object's outline in screen space is not the object's own geometry; the fix is to map the tapped point back into the object's local coordinates with the inverse of the object's transform and test there (Hughes and colleagues, 2013; W3C, 2019). The inversion is available because a rotation has determinant one, so the object's transform is invertible (Vince, 2012). The snapping is a representation problem. If the spin is interpolated through Euler angles or non-unit quaternions, the interpolation can cross a seam or leave the unit sphere, and Zhou and colleagues (2019) prove that every representation of three-dimensional rotations in four or fewer real dimensions is discontinuous, so no three- or four-number form removes the seam entirely; a continuous form needs five or six dimensions (Zhou and colleagues, 2019; Shoemake, 1985; Hanson, 2006). The strongest counter-argument is that neither fix is really mathematical. Argelaguet and Andujar (2013) report that no single selection technique is best and that performance depends on the task and the environment, so choosing the inverse-mapping test does not by itself settle the disambiguation; and Bowman, McMahan and Ragan (2012) argue that fidelity to the real world is a hypothesis to be tested rather than a default, so a smoother representation may not make the interaction better. The study would compare, on the same selection and animation tasks, a corrected and an uncorrected build, and would measure selection error rate, task time and the rate of interpolation artefacts, with the ambiguity of the test scene fixed in advance. A result in which the corrected build shows no improvement in error or artefact rate, or an improvement smaller than the run-to-run spread, would count against the team's diagnosis and favour the view that the defect is in the interaction design rather than in the algebra.

Eight participants cannot settle a 15 per cent claim on their own. If completion time has a standard deviation of about 20 seconds around a mean of about 120 seconds, eight participants give a standard error of 20 divided by the square root of eight, roughly 7 seconds, and a 95 per cent interval of about plus or minus 14 seconds. A 15 per cent improvement is 18 seconds, which lies inside that interval, so the new screen and the old one are not yet separated (Grinstead and Snell, 2003). The estimate is also flattered in its own arithmetic. Any usability measure computed from a small sample overestimates the problem-discovery rate, which makes a study appear to need fewer participants and to have found more problems than it has (Lewis, 2001). The strongest counter-argument is that the defect is the comparison, not the sample size. Hornbaek (2013) argues that experiments remain indispensable and that the remedy is a challenging comparison which can fail, with alternative explanations ruled out, so a small but well-designed study of a large effect could still persuade where a weak large one would not. The study that would settle the claim is a preregistered comparison of the two screens on the same task, with completion time, error rate and a satisfaction scale as measures, run at a sample size fixed in advance to detect the claimed 15 per cent at 0.80 power, and reporting an interval. A result whose interval straddled zero, or which failed to reproduce the 15 per cent, would count against the team.

The 700-millisecond figure is one point of a distribution, timed on one device against an idle server, so it leaves out load: the quantity that decides the wait a person meets is how many requests are in flight, which Little's relation ties to the arrival rate and the time in the system (Little, 1961). The single-server model makes the gap numerical. With a mean service time of 10 milliseconds the service rate is 100 requests a second, and the mean time in the system is 1 divided by (100 minus the arrival rate). At 80 requests a second, utilisation 0.8, the wait is 50 milliseconds; at 95 requests a second, utilisation 0.95, it is 200 milliseconds (Gross, Shortle, Thompson and Harris, 2008). A rise of 19 per cent in load has quadrupled the wait, and no reduction in render time offsets that at peak. The strongest counter-argument is that the delay a person experiences is constructed by the interface rather than simply caused by the pipeline. Elfleet and Chollet (2024) found that adding multimodal feedback to a conversational agent in virtual reality significantly improved presence, immersion and perceived response time without removing the delay, and Kurzweg and colleagues (2024) found that feedback latency changes the perceived weight of a button. What would settle the claim is a field comparison of the two designs at the 75th percentile of real visits, reported by device and connection, with perceived response time and task completion recorded alongside the render timing (Google, 2020).

The four problems are a requirement, a cascade conflict, a design failure and a security defect; each answer names the mechanism first. The zoomed layout fails Reflow: content must work at a width equivalent to 320 CSS pixels without scrolling in two dimensions. The fix is a single column or stacked grid below that width, with intrinsic sizing rather than fixed tracks. The evidence is a test at 400 per cent zoom on a 1280 pixel viewport, scrolled to check nothing is clipped, plus a keyboard pass on reading order. The card gap is the cascade: the vendor rule wins because the selectors weigh the same and order decides. The fix is to put the vendor sheet in a named layer and leave the library's own rules unlayered, since declarations outside a layer beat declarations inside one. The evidence is the computed margin on a card paragraph before and after the change. The form fails the error requirements: the message must be identified in text, and a correction suggested where one is known. The fix is a message beside the field, tied to the input so it is announced, with the typed value kept and focus moved to the field in error. The evidence is a screen reader pass that confirms the message is announced with the field, plus the typed value retained after a deliberate submission failure. The comment box is cross-site scripting: typed markup is being parsed as code. The fix is to write the value as text and encode it for the context, keeping line breaks by styling rather than by injecting markup. The evidence is a stored input containing a script tag that renders as characters rather than executing, tested in two browsers. Naming the test for each claim separates an audit from an opinion.

The recommendation should separate the two proposals, because they are not equally defensible. Automated auditing is a reasonable part of a process: it is fast, repeatable and catches large, mechanically detectable categories such as missing alternative text, missing form labels, empty links and low contrast. Its limits come from the tools themselves. The WebAIM Million tested a million home pages this way and found detected failures on the great majority of them, and its authors state that the absence of detected errors does not mean a page is accessible or conformant. A machine cannot judge whether alternative text is meaningful, whether an error message explains the problem, or whether a custom widget behaves correctly in the screen reader a particular person uses. The overlay is a stronger and weaker claim at once. Conformance means satisfying every success criterion, and the peer-reviewed evaluation by Hartman and Gorichanaz found that three AI-powered overlays corrected simple failures such as missing alternative text and unlabelled form fields, while serious failures remained and the overlays still reported the pages as fully compliant. I would keep automated checks as a regression net, retain manual testing with users of assistive technology on the paths people cannot avoid, and refuse the overlay until its supplier publishes a result that someone other than the supplier produced. The honest limits: that evaluation used two constructed sites and tested no disabled users, and Power and colleagues showed that conformance itself does not remove the problems real users meet. I would change my mind if an independent, peer-reviewed evaluation of a production site, tested with disabled users, showed conformance sustained after the overlay was installed.

A strong answer names one application, keeps to it, and describes two or three specific interactions rather than the product in general, because the marks reward a situated argument over a survey of features. State the three clauses first and test each against the application. Identification by address: does the site work from one address on any client, or does it push a distinct app for a named device, which is treating the platform as a device? Representation chosen from several: does the server send alternative representations, for example a text alternative or a reduced page, or one heavy page for all clients? Independent evolution: can the markup, styles and scripts change without breaking one another? The delegation argument then asks a harder question: because the author cannot know your device or connection, has the document been built so it still works when style or script fails, as Gustafson and Keith describe in layers? A site that shows nothing at all without scripting has chosen the device over the contract. Finally, propose evidence that could be collected: load the site with scripts disabled, on a slow connection and with a screen reader, and record what remains usable; or read the markup for a semantic document under the presentation. State what would count against your reading, such as the site working fully without scripting and serving reduced representations, which would show the contract being honoured.

A strong answer names one control, keeps to it, and inspects two or three specific behaviours rather than the whole product, because the marks reward a situated argument. Begin with the second tree: the browser builds the accessibility tree from the elements chosen, and a screen reader announces a role, a name and a state, so the first test is whether the control exposes all three. For an icon-only button drawn in CSS, that usually means checking whether an accessible name exists at all, since a label painted as a background image contributes nothing to the name. Then run the keyboard contract the authoring patterns state, which requires that the control be reachable with the Tab key, that Enter or Space activates it, and that a widget such as a combobox opens on a downward arrow, dismisses on Escape and accepts a highlighted suggestion on Enter. A native element arrives with that contract already met, while a custom control must reproduce it. The delegation argument then asks a harder question: because each property restored through ARIA depends on the script running, has the control been built so it still works when the script fails? Evidence that would show the reading is right includes loading the page with a screen reader and with the keyboard alone, and recording which of role, name, state and keyboard behaviour is absent. What would count against it is finding the control fully operable from the keyboard and correctly named, which would show that the element was chosen as an interface rather than as a picture.

A strong answer holds to one stylesheet and traces one or two conflicts rather than surveying everything. Start with the order of resolution: establish whether the conflicting declarations come from different origins, whether either is important, whether either sits in a cascade layer, and only then compare specificity and source order. A traced conflict might run as follows. A component rule sets a margin, a vendor stylesheet loaded later sets the same property with an equally weighted selector, and the value that renders is the later one because the two selectors score the same and source order decides. If the vendor sheet had been imported into a layer, the outcome would reverse, because unlayered normal declarations beat layered ones regardless of specificity. That trace is the argument. The evidence that would show the reading is right is operational rather than rhetorical: a browser's computed style panel showing which declaration won and why, and a second run after moving one sheet into a layer, which should flip the rendered value. What would count against the reading is finding no layer in use, no important declarations and no competing selectors, since then the stylesheet's order has not been arranged at all and the case says nothing about the contested question. On that question, the answer should take a side: a stylesheet disciplined by a naming convention but not by layers supports the convention position, while a stylesheet whose order is written down in layers supports the language position.

A strong answer holds to one arrangement and treats space and input as two separate questions. Start with the spatial description. A navigation bar that wraps into a menu below a certain width is responding to available space, and the width at which it changes is a measurement of a container or a viewport rather than a fact about hardware. Then ask the physical question the unit separates from it. A menu that reveals itself on hover assumes a pointing device that can hover and aim, and a horizontal strip that assumes a swipe assumes a thumb, so the arrangement carries an assumption about the input even when it never names a device. Name that assumption as an assumption, because one laptop can carry both a touch screen and a mouse, and the same page can be read with a thumb, a stylus or a magnifier. The smallest change is usually one modality query beside the width rule: enlarging the targets of the menu when the primary pointer is coarse, or replacing a hover-only reveal with a pressed state on a device that cannot hover. Then say what you would measure. The unit's evidence offers three options: error rate and time on a pointing task, following the target-size study; scrolling time and the number of scrolling actions, following the comparison of the two layout models; or, for a reader who zooms, whether an overview of the page survives magnification. Finish with the result that would count against you: if the unchanged arrangement performs as well as your version under the same input and in the same container, the modality query was not doing the work you claimed.

A strong answer separates the two variables before defending either. For contrast, the defensible source is the pair of polarity experiments: Buchner and Baumgartner found that colour contrast could not compensate for a lack of luminance contrast, so the quantity to secure is luminance contrast between the text and the background, and the value you keep can be checked by a tool. Buchner, Mayr and Brandt showed that the advantage of dark text on light is a luminance effect once the two display types are equally bright, which is an argument for keeping the page bright rather than for a particular hue. The honest label for that value is a threshold: the contrast criterion a standard enforces is checkable, and Yu and colleagues show it is not sufficient, since the luminance a reader needs on a tablet depends on the ambient illumination and not on the character contrast ratio alone. For line length, the honest label is a range. Dyson and Haselgrove found that 55 characters per line produced the highest comprehension and was read faster than short lines, but Shaikh and Chaparro found online news was read fastest at 95 characters while satisfaction did not differ, and Cerepinko and colleagues judged 79 characters and above readable enough on a tablet. Those are different measures pointing different ways, so a defence that claims 55 is the correct number is overclaiming. The better defence states the task the page is for, chooses a value inside the reported range, and says which outcome it is optimising. Then name a measurement: reading rate and comprehension by question type on your own content, or preference between two versions, and state the result that would count against you, which is that the version you rejected performs at least as well as yours on the same measure with the same readers.

A strong audit separates the four stages before judging any of them. For the labels, record whether each field has a label that stays on screen, because that is what gives the field its name, and say so with the evidence that labels are frequently missing or unassociated with their controls (Islam et al., 2010). If the form relies on placeholder text inside the answer box, note that the placeholder leaves when typing begins and that the experiment on placeholder text found longer prompts produced longer but less substantive answers with no benefit from short ones (Kunz et al., 2020). For prevention, record what the form tells you before you answer: whether a format rule is stated in advance without a distracting example (Bargas-Avila et al., 2010, Guideline 13) and how required fields are marked, remembering that the coloured marking beat the customary asterisk on errors, speed and satisfaction (Pauwels et al., 2009). For recovery, record the wording of the message, whether the field you filled is cleared, and whether the problem is visible at a glance (Bargas-Avila et al., 2010, Guidelines 14, 15 and 17), and note that the chapter itself marks the never-clear rule as expert practice rather than measured evidence. For validation timing, both positions are available: the guideline chapter puts errors at submit on the strength of an experiment in which inline messages produced more errors, while the practitioner study of 22 users found inline validation improved every measure and that validating on blur beat the interrupting versions (Wroblewski, 2009). Say which you follow and give your reason, which a good answer grounds in the content of the form, for example that the fields are ones a user cannot answer confidently first time. Then name one change and the measurement that could falsify it: two versions of the same form, completion time, errors, abandonment, and whether the messages were read.

A strong answer keeps the instrument and the user apart. For the automated part, name the tool and report what it found, then state the unit's warning: Vigo, Brown and Conway (2013) benchmarked six tools and found coverage of at most 50 per cent of success criteria, completeness between 14 and 38 per cent and correctness between 66 and 71 per cent, so a clean report is evidence about the tool as much as about the page. Add that Kelsey-Adkins and Thompson (2022) found very poor agreement between four command-line checkers, and that the WebAIM Million (2026) audits a million home pages yet its authors still say the absence of detected errors does not mean a page is accessible. For the manual part, record what happened when you tried the task, and say for each barrier whether a WCAG success criterion covers it. This is where Power and colleagues (2012) apply: across 32 blind users and 16 websites, only 50.4 per cent of the 1,383 problems logged fell under a criterion, so you should expect at least some of what you meet to lie outside the list, and you should say so rather than force it into a criterion. The judgement then falls out of the evidence: a page can pass the checker and still fail the task. State both findings separately, as Olalere and Lazar (2011) do when they contrast a compliance claim with an inaccessible site. If you propose a change, name the measurement that would falsify it: the same task, the same kind of user, the same request, before and after, with the barrier recorded and something observable such as completion or time to complete. Conformance alone will not settle it, which is the point of the unit.

A strong answer separates three things. First, state the target precisely, in the form the guidance uses: a metric, a threshold and a percentile, judged separately for mobile and desktop. If you choose Largest Contentful Paint, the threshold is 2.5 seconds and the question is whether the 75th percentile is the right line for this audience (Google, 2024). Justify it as a distributive choice rather than a preference: the median would describe the typical session, while the percentile forces you to say how bad things may be for the slow quarter. Second, name that slow quarter concretely and say what evidence you would collect. The field data give you a starting assumption, because page weight grew again in 2024 and the Web Almanac names the performance inequality gap, on the argument that heavy pages fall hardest on users with cheaper devices and metered connections (HTTP Archive, 2024). Add the measurement side: Kelton and colleagues (2020) show that existing load metrics were built for desktops and that mobile users view several viewports during a load, so a desktop measure may misdescribe exactly your slow quarter. Salutari and colleagues (2020) go further and show that perceived load time is multi-modal and that one metric cannot say whether a page loaded fast, which is the source that should make you state your target as a proxy rather than as the experience. Third, answer the question and give a falsifier. Say who the target is for and what would change it, and be honest about the counter-argument: Shiau and Luo (2013) found continuance intention predicted by involvement, satisfaction and perceived enjoyment rather than by speed, and Bouch, Kuchinsky and Bhatti (2000) found judgement depends on expectation and context. A target that meets its number is therefore necessary and not sufficient, and a strong answer says so.

A strong answer separates the three parts rather than blending them. First, name the place where stored content re-enters the document and say which parser receives it: a comment written into an HTML context is parsed as HTML, so the encoded output must be entity encoded, whereas the same value written into an attribute, a URL or a JavaScript context needs different encoding, because browsers parse each by different rules and the same string can be safe in one and executable in another (OWASP Foundation, n.d.). Second, name a defence and test it against evidence on both sides. If you choose contextual encoding, Klein and Johns (2024) limit your confidence by bypassing nine of eleven sanitisers through parsing differentials, which tells you that the defence is only as good as its agreement with the browser's parser. If you choose a nonce-based policy, Weichselbaum and colleagues (2016) supply the case against its predecessors, with bypasses in 94.72 per cent of distinct policies, while Pan and colleagues (2016) show that generating and enforcing a strict policy without server changes is feasible. If you embed a third-party script, note that Duan and colleagues (2021) found 339 previously unknown malicious packages in the registries an author draws on, which makes verification of included content a live question. Third, answer the contested question for your own case and give a falsifier. If you argue the toolchain should carry it, cite Pan and colleagues (2016) and the finding of Ren and Yue (2024) that most sites lacking a policy could deploy a safe one. If you argue the designer carries it, cite Khodayari and Pellegrino (2023), whose DOM clobbering attacks need no script tag and therefore no policy about script sources, and say what you would measure to change your mind.

A strong answer separates the three parts rather than blending them. First, name the elements the page would declare and the inference the alternative requires. A product listing that marks up each item with a heading and a structured description gives a reader named fields, whereas a listing built from nested dividers leaves the record structure to be recovered from visual formatting, which Grigalis (2013) shows is possible by clustering visually similar elements and deriving reusable wrappers, at the cost of fragility. Second, take a position on the shared good and test it against evidence both ways. If you argue structure is shared, note that declared meaning is what assistive technology already needs and that Song and colleagues (2025) treat machine generation and human accessibility as the same task of describing an image well, and that Kim and Ji (2025) show the interaction design of a generative interface changes what a person learns. If you argue it is traded, cite Khosravi and Yoganarasimhan (2026), whose difference-in-differences estimate puts the fall in English search referrals under AI Overviews at 5.45 per cent and 4.82 per cent, Lin (2026), who finds a 4.3 per cent ranking penalty for opting out of AI search, and Wadinambiarachchi and colleagues (2024), who find that a generative tool in ideation reduced the variety and originality of ideas. Third, answer the contract question for your feature and give a falsifier. If you claim continuity, cite ActivityPub (W3C, 2018) as a 2018 Recommendation predating the agent wave; if you claim a new contract, say what would be true of agent success on pages like yours if you were wrong, for instance no measurable difference between declared and undeclared versions of the same page.

A strong answer holds to one alert and reasons from evidence rather than from preference. First, the alert must not rely on colour alone: it should pair any change of hue with a label and an icon, because colour is an opponent-channel construction and a user whose red-green channel is weak would otherwise lose the signal. That evidence is strong, since it rests on the physiology of colour vision. Second, the alert should be salient and, where the setting allows, announced on a channel the monitoring task does not already occupy, because inattentional blindness shows that an unexpected event can pass unnoticed while attention is engaged. The phenomenon is robust, and the detection rate moves between samples, so the design should not be justified by one quoted percentage. Third, the acknowledgement should be designed for the first hour and for the thousandth hour. A clinician new to the unit reads every alert, while an experienced one answers by habit, so the control should separate a new alarm from one already cleared and keep a way back: that follows Reason's distinction between slips and mistakes, and Norman's account of habit intruding into a routine. The practice evidence also warns that an expert's advantage is knowledge of real cases rather than a larger store, so training on real alarm patterns transfers further than drill on invented ones. A strong answer finishes by naming the evidence that would test the design, such as a detection check that adds a competing task and a count of acknowledgements made without reading.

The conclusion does not follow. Equal task times establish equal effectiveness on the measured task, and nothing more. McCarthy and Wright argue that the felt experience of technology is emotional, intellectual and sensual at once, and that people live with technology rather than merely use it, so an account built only on task performance omits part of what is being judged. Two systems can take the same time while one leaves the user anxious and the other confident, and the difference matters for whether the system is trusted, returned to and recommended. Affect functions as information about the situation, which means the felt state is itself a judgement rather than a decoration. A better study would add measures that reach the affective and meaningful side: a short self-report of arousal and valence after the task, interviews that ask what the session felt like, and, where use is spread over time, an experience-sampling diary. These methods are weaker and less standardised than task timing, and two teams can reach opposite conclusions from them, so the claim they support should be stated with that caution. The revised conclusion is that the systems are equally fast, and that their quality is still to be established.

A strong answer names one screen and holds to it, describing two or three specific colour and contrast decisions rather than summarising the product, because the marking rewards a situated argument. Separate the two kinds of claim the unit distinguishes. A contrast choice between text and background can be defended from measurement: a ratio can be checked, and the physiology of opponent channels and contrast sensitivity tells you that legibility rides on luminance contrast rather than hue, so a figure such as the standard's 4.5 to 1 for ordinary text and 3 to 1 for large text is a floor you can test against. A choice to use red for error and green for normal can only partly be defended this way, because the red-green channel can be weak or absent, and the sources show that deficiency is a normal fact of the population rather than a rare exception. That is why the redundancy rule follows from evidence: pair hue with shape, position or a word, so the meaning survives a failed channel. A choice of a particular palette for its mood, by contrast, is a preference, and Hall and Hanna show that preferred colours raise aesthetic ratings while not necessarily being the combinations that read best, so taste and readability can pull apart. The second half is what separates analysis from assertion. Name a test that could actually settle it: a reading task that holds contrast constant while varying the surround and the letter size, measuring reading rate rather than detection, to see whether the screen sits on the plateau or the cliff; or a check of the scheme through a dichromat simulation and with readers who have named deficiency types, to see whether the redundant code still works when red and green collapse. Then say what result would count against you: if a hue-only code proved readable across the deficiency types you tested, the redundancy rule would be unnecessary for that screen, and if the contrast ratio sat on a steep part of the curve in your users' conditions, your floor would be too low.

A strong answer names one screen and stays with it, describing two or three specific layout decisions rather than summarising the product, because the marking rewards a situated argument. Separate the two strengths of claim the unit distinguishes. An encoding decision can be licensed by measurement: Cleveland and McGill (1984) ordered elementary perceptual tasks by accuracy, with position along a common scale at the top and colour hue at the bottom, so a choice to encode a key quantity by position, or a decision to replace a pie chart of several categories with a dot or bar chart, can be defended from measured error on the same judgements rather than from preference. A decision to make one element stand out can be defended from Wolfe and Horowitz (2004) only if the difference is carried by a guiding attribute such as colour, motion, orientation or size; a design that relies on a difference in an arbitrary shape or an icon detail is not supported, because those are not guiding attributes, and the prediction is that the element will not pop out. A decision about how to group related controls, by contrast, is closer to a hypothesis. The gestalt principles describe how marks bind into objects (Wertheimer, 1923; Wagemans et al., 2012), but they state a direction and not a size, so a chosen spacing, border or common region is a prediction that the grouping will read as intended. That prediction is weaker still because grouping is conditional and depends on the observer: Kimchi (1992) shows the global advantage is not unconditional, and Koc and colleagues (2025) show the orientation bias in grouping by proximity depends on working-memory load, so the same layout may group differently for a user under memory load. The second half of the answer separates analysis from assertion. Name a test that could be run: for the grouping decision, a timed task in which users locate a target within the grouped panel, with the grouping manipulated and working-memory load varied, would show whether the grouping survives the load; for the search decision, a response-time-by-set-size measurement would show whether search for the highlighted element is efficient or serial, since an efficient search gives a shallow slope as items are added. Then say what would count against the reading: if the element that differs only in an arbitrary icon detail were found as quickly as one that differs in colour, the appeal to the guiding-attribute list would be wrong for that screen; if the grouping held up under memory load, the caution drawn from Koc and colleagues would not apply here.

A strong answer stays with one named screen and describes two or three specific ways a change or an alert could be missed, rather than listing general advice. Start from the mechanism the unit establishes. Change blindness occurs when a brief disruption coincides with a change, and O'Regan, Rensink and Clark (1999) showed that six small high-contrast shapes which did not even cover the change were enough to hide it, so a dashboard that refreshes a value while the screen reflows has supplied its own disruption. That failure is licensed by evidence. Inattentional blindness is likewise licensed: Simons and Chabris (1999) found that detection depended on the unexpected object's similarity to the tracked items and on the difficulty of the primary task, and not on how close it came to the attended locations, so moving a warning next to where the user is looking does not save it. Wallisch and colleagues (2023) add that speed, not exposure or salience, drove detection across 4,493 participants, so an alert that appears without movement is the weaker choice. Most and colleagues (2001) supply the boundary: a bright red cross with a unique colour, shape and motion path was still missed by nearly 30 per cent of observers when the task supplied a rule for ignoring it, so distinctiveness is not a guarantee. The hypothesis to flag is the claim that a particular animation or banner will be noticed on this screen, because none of these studies tested this interface, and the unit's treatment of the effect as conditional supports treating such a claim as a prediction rather than a result. The design rule follows from the evidence rather than from taste: put the alert on a channel the primary task does not occupy, mark the change with a visible transient, and do not rely on colour or size alone. The test must be able to fail. Ask a sample of users to complete the screen's primary task, introduce the alert as an unexpected transient during that task, and then probe for it with a question that guards against a conservative criterion: a forced-choice question about its location or colour, with occasional absent trials in which no alert appears, following Nartker and colleagues (2024). If detection is at chance once the reporting criterion is corrected, the design claim is refuted; if the alert is reported reliably above the false-alarm rate, the claim survives.

A strong answer holds one named setting throughout and keeps evidence and prediction apart. Begin from the account of interference being assumed, because the unit's contested question is exactly this. If the design assumes the multi-resource model (Wickens, 2002, 2008), then the claim to defend is that the incoming task shares a dimension with the primary task, for example that a spoken alert competes with a spoken clinical instruction for the auditory modality, and the licensed move is to move the alert to a dimension the primary task leaves free. If the design assumes the bottleneck account (Pashler, 1994), the claim is that both the interruption and the primary task need action selection, so no amount of modality separation will remove the cost, and the licensed move is to defer the interruption rather than re-route it. State which you assume and why, and note that Salvucci and Taatgen (2008) show a threaded model can reproduce both patterns, so a fit to data is not by itself a discrimination. The licensed resumption claims come from the memory-for-goals work: Altmann and Trafton (2002) make the reactivation of a decayed goal the mechanism, Trafton and colleagues (2003) show that a warning before the interruption speeds resumption, and Monk and colleagues (2008) show that demand matters as much as duration, so a shorter interruption is not the remedy on its own. The design follows: record where the user stopped, state the next step, and make the suspended task easy to re-establish. What remains a hypothesis is the claim that this particular screen will behave this way, because none of these studies tested it. The test must be able to fail. Interrupt a sample of users during the primary task, measure the resumption interval as Iqbal and Horvitz (2007) did in the field, and measure the affective cost as Bailey and Konstan (2006) and Mark and colleagues (2008) did, since Mark and colleagues found that output alone can look unaffected while reported stress rises. Say in advance what result would count against your design, for example no reduction in resumption time against a no-cue control.

A strong answer holds one screen throughout and separates what the sources license from what remains a prediction about this screen. Begin with the account of the limit you assume, because the unit's contested question is exactly that. If you assume the discrete-limit position (Luck and Vogel, 1997; Cowan, 2001), the licensed claim is that only a few integrated items can be held while the other task runs, so a checkout that makes the user carry a code across a device boundary is asking for an item that will often be lost, and the licensed remedy is to bring the item onto the same screen. If you assume the resource position (Ma, Husain and Bays, 2014), the licensed claim is weaker and more graded: what falls as load rises is the precision of each held item, so the failure appears as a mistyped digit or a wrong field rather than a blank one, and the remedy is to raise the fidelity available to the item that matters. If you assume the attention-control position (Engle, 2002), the licensed claim is that capacity is spent by competing responses, so a screen that offers several plausible options while the user holds an item costs capacity even though no extra item was added, and the remedy is to remove the competing response. State that the average of about four is a summary of a distribution rather than a property of your user, since Vogel and Machizawa (2004) found capacity ranging from about 1.5 to about 5 objects. The design follows from Zhang and Norman (1994), who showed that changing the external representation changes the task, and from Kirsh and Maglio (1994), who showed that people act on the world to make thinking cheaper. What remains a hypothesis is that the redesign helps on this screen, because Scaife and Rogers (1996) argued that the benefit of a graphical representation is often assumed rather than shown. The test must be able to fail: measure the rate at which the carried item is entered correctly with and without the change, and register in advance what difference would count against your design.

A strong answer holds one interface and one experienced user throughout and keeps evidence apart from inference. Begin with what practice is claimed to change. Chase and Simon (1973) and Gobet and Simon (1996) license the claim that the practised user perceives structure the beginner does not, so the objects on the screen are not the same objects for the two users, and Chi, Feltovich and Glaser (1981) license the claim that the practised user groups the material by principle where the beginner groups it by surface feature. Fitts (1964) and Anderson (1982) license the claim that attention a step consumes at the start is released later, and that the release is capacity the practised user has available for something else. From those claims, the guidance you would present at the start is whatever carries the search the beginner cannot yet do, for example a labelled route through the record, a worked example of the common task, or an explicit statement of which of two similar fields to use. The fading rule follows from Kalyuga, Ayres, Chandler and Sweller (2003), who report that support which helps novices can lose its value and harm experienced learners once it becomes redundant, and from Sweller (1988), who argues that unguided search spends capacity that would otherwise build schemas. Say plainly that these studies concern instructional materials, so their extension to a working interface is an inference, and that the unit's own contested question turns on exactly this point: the evidence for a fading interface is an interaction between support and expertise, not a main effect of support. The test must be able to fail. Within one study, compare novices and practised users on the same screen with and without the support, measure the task that matters, and report the interaction and its size. If the support helps both groups, or neither, the fading argument is not supported, and the honest outcome is to leave the screen as it was.

A strong answer holds one interface and one decision throughout, and keeps evidence apart from inference. Begin by naming the default precisely: which option is preselected, and what the user must do to leave it. Then state what the evidence licenses. Johnson and Goldstein (2003) license the claim that a preselected option changes the outcome, since consent ran from about 4 to 28 per cent in their opt-in countries and from about 86 to 100 per cent in their opt-out countries, and Thaler and Benartzi (2004) license the claim that a commitment default can hold over time, with the average saving rate rising from 3.5 to 13.6 per cent across 40 months. Tversky and Kahneman (1974) and Kahneman and Tversky (1979) license the weaker claim that an anchor shifts an estimate and that a loss-framed description is not the same as a gain-framed one, and Stanovich and West (2000) license the claim that the size of that shift varies between people, so a single figure describes no one in particular. Say plainly what is inference: the size of the effect on your interface is not given by any of these studies, and Gigerenzer (1991) and Gigerenzer and Brighton (2009) warn that an effect may be a property of the format you happen to have chosen. Defend the choice on principle by asking whether you would accept the default if the user read it as your recommendation, and whether a reasonable person who missed the setting would be worse off. The test must be able to fail: randomise users to the default and to its opposite, measure the decision that matters, report the effect with its variation rather than as a single percentage, and state in advance what result would show the default makes no difference, in which case the honest outcome is to change it.

A strong answer holds one interface and one failure throughout. Begin by naming the failure precisely and classifying it with this unit's sources. A nurse who selects the wrong patient from a list because a familiar surname captures the action is committing a slip: the intention was proper and the result did not conform to it (Norman, 1981), and the failure sits in the activation and triggering of a habitual routine rather than in the goal. A clinician who enters an out-of-range dose because the guidance was misread is committing a mistake: the intention itself was wrong, and Rasmussen (1983) would place the failure at the rule or knowledge level. Then state the remedy that fits the mechanism. The slip is answered by a constraint in the interface, a distinct identifier on the row and a confirmation that names the patient; the mistake is answered by information and a check, a dose range beside the field and an independent verification (Norman, 1983; Reason, 2000). Say plainly what is inference: the sources license the mechanism and the direction of the remedy, not a measured reduction in errors on this ward. Name the counter: Perrow (1984) holds that in systems both interactively complex and tightly coupled, failures interact and propagate before anyone can intervene, so a constraint or a check may not be enough, and the honest options then include redesigning the task itself. A strong answer ends by naming what evidence would test the claim, for example a before-and-after count of the failure with the conditions behind it measured rather than assumed.

A strong answer uses the law to rank the options rather than to promise a number. Fitts (1954) showed that movement time rises with the logarithm of distance over target width, so on the dashboard a frequent control is quicker to reach when it is larger and closer to the pointer's resting position, and the ranking of two candidate layouts follows from the law. The phone is where the answer must change register. Chakraborty and Yadav (2022) found the touch literature small, with positive, null and negative results and no consensus, and Zhang (2023) found a periodic effect of movement direction on a touch screen that the standard index of difficulty does not capture, so a target reached by a thumb travelling across the screen is not the case the mouse constants describe. Accot and Zhai (2003) add that a two-dimensional target is not simply a wide one, since for width to height ratios between one and two the amplitude of the movement dominated the directional constraint. What a team should therefore state is that the law fixes the direction of the effect, larger and nearer is faster, and that the constants must be measured on the device, the grip and the directions that matter. Name the strongest counter: the law has already transferred once, because Card, English and Burr (1978) found the mouse fitted by Fitts's law with a slope close to the best seen in other eye-hand tasks, so transferring it again is a matter of re-estimating constants within a standard protocol rather than an overreach, which is what Soukoreff and MacKenzie (2004) recommend. The counter to the counter is that a standard protocol makes constants comparable, it does not make them portable. What would settle it for this team is a within-subjects test of the two layouts at the real sizes, on the real device, with the thumb directions the app uses, reporting movement times, error rates and end-point scatter; the range of difficulties should be varied on purpose, as contemporary practice requires, so that the slope and intercept are identified rather than assumed.

Start by stating the claim with its instrument and its theory. The detector reads rapid taps and abandoned forms, which is behavioural data, and it labels the result frustrated. That label comes from a categorical account in which frustration is a state a machine can identify, and the unit shows that this account is contested: Barrett (2006) finds the evidence against boundaries carved in nature accumulating, and Lindquist and colleagues (2012) find no consistent one-to-one mapping between discrete categories and brain regions. A dimensional reading would report arousal and valence instead (Russell, 1980), and Bradley and Lang (1994) show that a dimensional measure such as the Self-Assessment Manikin operationalises that theory rather than testing it. So the detector's label is a measurement claim, not an observation. The strongest counter is the interactional one. Boehner and colleagues (2007) argue that treating affect as information reproduces the information-processing model, that emotion is made in interaction, and that a system should help people interpret their own emotion rather than detect a state; on that view the detector should be replaced by a form that shows the user what happened and why, and the human handover is a design choice rather than a measurement. A middle course keeps the behavioural signal as a prompt to check rather than a verdict, and adds a second channel, since Mandryk and colleagues (2006) show that physiology can be set beside self-report and can disagree with it. To settle whether the detector works, state what would falsify it: the label should predict what the adviser finds when the user is asked, above the rate expected from abandonment alone, and a brief self-report measure should agree with the label rather than diverge from it. Report the agreement, the disagreement and the share of users for whom the two channels disagree.

Start by refusing the question as posed. 'Good' is not measurable, so convert it into two answerable questions: what are the common tasks students attempt with the app, and where does each task fail? That names a population, a behaviour and a decision, and it fits the access available. Use two methods deliberately. Run six semi-structured interviews about the last occasion the participant used the app, asking what they were trying to do and what happened, never inviting them to rate the product in the abstract. Then observe six participants attempting two or three representative tasks in their own setting, because a task described smoothly may collapse in the room. A single session still loses the events that matter most, such as the panicked search in a corridor and the second attempt the next morning. Add a one-week diary for four of the twelve, designed for compliance with one prompt a day, short entries, and a rule for the days an entry goes missing. Recruit with a screener that reaches non-users as well as heavy users, since an app that heavy users like may be one that others avoid, and include at least two participants who use assistive technology or an older phone. Ethics: written informed consent naming the recording, the storage, the retention period and the right to withdraw up to the point of analysis; no manager or tutor recruiting their own students; an incentive proportionate to a student's time. Define the evidence in advance. The client should change the decision if observation shows repeated failures at one task, or if a pattern of workarounds points at a missing function. If the interviews and the observation disagree, do not average them: report the divergence as the finding, and say which method answers which part of the question.

The report makes three separate errors. The first is a sampling error: four hundred respondents are the people who chose to answer the survey, so self-selection makes the figure a description of the sample rather than of users in general. The second is a measurement error. A question that invites people to say what they want is an attitude item, and attitudes are subject to acquiescence, the tendency to agree with whatever is proposed, and to social desirability, the tendency to give the answer that looks reasonable. The third is an error of inference: the focus group's agreement is not independent confirmation, because group members influence one another and a dominant voice can carry the discussion, so two methods with the same weakness do not triangulate. The evidence can support a claim about which features this sample named. It cannot support a claim about what users need, and it cannot rank a want against an observed problem. Replace it with a study that measures behaviour: instrument the product, or observe a small sample attempting the tasks the features are meant to serve, then ask how often each missing feature is implicated in an actual failure. That study can be small and can be run in a fortnight, and unlike a poll it produces a finding the team is obliged to act on rather than one it can simply endorse.

A strong answer narrows the library's request into a decision before choosing a method. The vague question, why students abandon the system, is not answerable; a decision-first version names one decision, for example which step of the reading-list flow the library will redesign, and one observation that would settle it, for example the recorded drop-off point across a term and what students do immediately after leaving. State that the observation is fixed before collection, because a threshold chosen after the data is chosen to fit the data, and say what result would count against the claim. Then choose a method that follows from the question rather than from a menu: a survey can measure how many reach each step, but it cannot explain why, so an observation of a few students working in the setting is the method the question points to, with its own stated limits about who is observed and how many. Use Greenberg and Buxton to test whether the study should run at all: if the library will not act on the drop-off point, no decision turns on the answer and the honest conclusion is to decline the study. Finally, engage the contested question rather than assuming the unit's side. A made artefact, such as a redesigned prototype, might disclose a problem the fixed question never anticipated, which is the research-through-design position that a method can generate a question. So the answer should say what finding would change the plan: if the observed students reveal the abandonment is caused by something the chosen question never named, the question was wrong and a generative study is warranted; if the drop-off point matches the prediction and no decision awaits it, the study should not run.

A strong answer separates the two things the team has fused. The advertised sample reaches parents who use that platform, see the advert and choose to answer for a voucher, so the five hundred are a self-selected subset of a frame the team has never defined, and the abandonment rate they produce describes that subset rather than the families the council serves. The population should be named first, together with the parts of it the advert cannot reach: parents without the platform, parents who abandoned long ago and see no reason to return, and parents whose first language is not the one the form uses. Size does not repair this, which the Literary Digest case shows, because the 1936 poll had nearly 2.4 million returns and still forecast the wrong winner, and Lusinchi's reanalysis puts most of the error at the difference between respondents and nonrespondents. The team would have to report its frame, its recruitment route, its response rate, and the known differences between those who answered and the local population on the variables that matter, such as language, device and entitlement to free school meals. Because the standard is a benchmark rather than an assurance, Baker and colleagues (2013) require the assumptions behind a non-probability sample to be stated with evidence of how departures from them affect the estimate. One observation would settle whether the failure runs in a direction: compare the abandonment rate in the self-selected group with the rate the council's own server logs record for all users over the same period. If the interviewed parents abandon less often than the logs show, the sample has missed the parents who gave up, and the error runs one way.

The two studies need two different rules, which is the unit's central claim. The interview study builds an account from talk, so its number should come from a saturation argument rather than from habit, and the team should say in advance what would count as a new theme and how it will decide that new themes have stopped appearing. Guest, Bunce and Johnson (2006) tracked sixty interviews and found most themes present within the first twelve, and Hennink and Kaiser (2022) put measured saturation commonly at nine to seventeen interviews, so twenty may be generous, but generosity is not the point: the point is that the figure must be tied to a stated rule and the conditions that rule assumes, which for a more heterogeneous group of patients could push the estimate above the range. The comparison study makes a quantitative claim, so its number must come from a power calculation, not from a common figure. The team should fix the smallest effect worth detecting, the five-point reduction it names, choose an alpha and a power level, and derive the per-condition sample from those, following the conventions Cohen (1992) set out. Thirty per condition may be far too few if the effect is small relative to the natural variation in missed appointments, and Button and colleagues (2013) show that underpowered work reports distorted effects rather than merely failing to find them, which is why the calculation is part of the claim. For each study the report must give the population, the rule, the assumptions the rule rests on, and the uncertainty in the number, which is what Borsci and colleagues (2013) demand when they say a small figure is acceptable only within a stated model whose uncertainty is reported. The result that would show the interview study stopped too early is a late interview producing a theme, or a whole category of reason, that no earlier session had raised and that changes how the service should be redesigned, because that would demonstrate the stopping rule had fired before the range of experience had been covered.

The interview can establish what residents say about the form and the conditions in which they say it, and it cannot establish by itself what they actually did at the screen. The team should therefore plan to observe or log the abandonment as well, and treat the interview as the account of it. The question 'Why did you abandon the form?' asks for a cause, and Nisbett and Wilson (1977) found that people have little direct introspective access to such processes and answer from a priori causal theories, so the reply will be coherent and may not match what governed the behaviour; Wilson and Dunn (2004) add that self-knowledge often improves only through indirect routes such as observing one's own behaviour. It should be replaced by a request for an episode: 'Tell me about the last time you stopped part way through, starting from when you opened the page.' 'How confusing did you find step three?' is leading, since it supplies the verdict the team hopes to hear; it should become 'What were you trying to do at that point, and what happened next?' 'Would you use the form again if we simplified it?' asks for a forecast, which the unit rules out, and should ask instead what the resident did on the last occasion and what they did next. To keep reports near heeded processes, the team should ask concurrently where possible, following Ericsson and Simon (1980), who argue that verbal reports are data only where the mechanism generating them is respected and that concurrent verbalisation of what is being heeded is defensible. The medium belongs in the method: Lo Iacono, Symonds and Brown (2016) show that remote interviewing changes rapport, non-verbal cues and ethics, so the team should record how each interview was conducted and treat remote and face-to-face accounts as differently produced rather than interchangeable. Reflexivity is part of validity, following Roulston (2010) and Denzin (2001), so the guide, the interviewer's role and the setting should be reported. The counter-test is a criterion-validity check: log where residents actually abandoned the form, then ask them concurrently and retrospectively, and compare each account with the log, since Atkinson and Silverman (1997) and Nunkoosing (2005) warn that an account can be shaped by the encounter rather than by the event, and only a comparison against behaviour can show which account the study should trust.

The study can establish how medication work is organised in that ward, including its informal practices, exceptions and dependence on other people, and it can build a work model from what was seen rather than from what nurses say about it, following Beyer and Holtzblatt (1998). It cannot establish what medication work is like across the trust, because one ward is one site: Rogers and Marshall (2017) set out the trade of lost experimental control for ecological validity, and a methods section should state that the claim is bounded rather than general. It also cannot establish whether a redesigned system will work, since the study is not a test of one. Reactivity cannot be handled by claiming to correct for a known effect. McCambridge, Witton and Elbourne (2014) reviewed nineteen purposively designed studies, found twelve with some statistically significant evidence of an effect, and concluded that there is no single Hawthorne effect and that little can be securely known about its conditions, mechanisms or magnitude. The plan should therefore record whether nurses altered routines for the visit, how the observer's presence was received, and whether a second observer saw the same practice, treating those as findings rather than noise. Adair (1984) supports this: participants define the situation for themselves, so the nurses' reading of the study is what must be described, and Levitt and List (2011) show that the canonical example is itself contested, having found the familiar descriptions of the illumination experiments fictional. The single site limits the study to an account of this ward, so the plan should sample deliberately within it, choose shifts and handovers that carry the variation, and say what the ward cannot support a claim about. Finally, the team should state which purpose it is serving. If it follows Randall, Harper and Rouncefield (2007) it must commit to the analytic work that turns notes into a design decision; if it follows Dourish (2006) or Crabtree and colleagues (2009) it should say that the value lies in the analytic account of the setting and refuse to pretend that a bulleted implications list was the point. Two days of shadowing two nurses is also a small sample that should be justified rather than assumed, and the notes or video should be chosen deliberately, since Heath, Hindmarsh and Luff (2010) show what written records lose.

The momentary instrument establishes what happened close to the time it happened. Shiffman, Stone and Hufford (2008) present repeated sampling in real time and in the participant's own environment as the way to minimise recall bias, and Redelmeier and Kahneman (1996) show why that matters: retrospective totals tracked the peak and the final three minutes of a procedure rather than its duration, so a later account cannot be used to estimate how often a reminder was missed or how long a lapse lasted. The reconstruction interview establishes something different, namely how patients make sense of the day as a whole. If the team wants the frequency of missed doses, it needs the momentary record; if it wants the patient's account of why a dose was missed, the interview is the instrument, and Kahneman and colleagues (2004) show that reconstruction can reproduce aggregate patterns of time use and affect at lower burden. The plan should therefore not treat one as a cheaper substitute for the other. Compliance has to be designed and reported, not assumed. Jones and colleagues (2019) pooled 126 studies and found mean compliance of about seventy-five per cent, below the recommended rate and unrelated to prompt frequency or assessment length, so shortening the schedule will not fix it. The team should follow the terms Stone and Shiffman (2002) set out: state the sampling rationale, the prompt interface, the training and monitoring of participants, the analysis plan and the rate of compliance, and it should verify the timing of entries rather than accept self-reported compliance, because Stone and colleagues (2003) found reported compliance of about ninety per cent against eleven per cent verified when the timing was checked with a photosensor. If three prompts a day for three weeks is too demanding, the honest alternative is to sample fewer days well than to collect entries that were backfilled. The measurement changes what it measures, and this has a design consequence. Czerwinski, Horvitz and Wilhite (2004) name the influence of journaling on the events journaled a Heisenberg-style challenge, and Ho and Intille (2005) show that when a prompt arrives changes how it is received. A prompt about medication arriving during a consultation is an interruption, and the plan should say so, bundle the prompts to moments the patient nominates, and treat the observation that prompting made patients more aware of their own adherence as a finding rather than noise. The team should also pilot the burden before the study proper. Finally, the plan should state whether momentary sampling is necessary. If the decision turns on how often doses are missed and when lapses happen, the reconstruction interview cannot carry it, and the three-week prompt study is justified. If the decision only turns on how patients describe the reminder, the day reconstruction method may be sufficient. Whichever is chosen, the report should state the sampling scheme, the verified compliance, the effect the prompts appeared to have on behaviour, and the bound of what a three-week sample can support.

The reported alpha does not settle reliability on its own. Cronbach (1951) derived alpha as the mean of all split-half coefficients and warned that a test divisible into distinct subtests should be divided before the formula is used, since it then estimates the concentration of the first factor rather than the structure of the whole. This scale is visibly multidimensional, mixing satisfaction with ease of use, so a single coefficient hides the structure. Sijtsma (2009) argues further that alpha is unrelated to the internal structure of a test and cannot equal the reliability of the scores under the usual assumptions, so the team should model the items and report a defensible estimator. McNeish (2018) offers the constructive route: the criticisms target alpha rather than reliability, so report alternatives such as omega total and state the assumptions each makes. The instrument should therefore be split into its two constructs and each reported separately. Wording and format are design decisions with measured consequences. Schwarz (1999) shows that the same construct asked two ways yields systematically different answers, so each item should be reviewed for wording, format and context rather than accepted as a neutral question. Krosnick (1999) reports that the verbal labels on rating-scale points shift the answers, so the scale labels should be fixed and justified, and Krosnick (1991) predicts that hard or double-barrelled items raise satisficing, so items that ask two things at once should be rewritten. Response set must be designed against, not merely observed. Nederhof (1985) treats social desirability as a measurable bias with detection and prevention methods and found no single method sufficient, so the team should combine detection with prevention, for instance self-administration and forced-choice items. The flagged twelve per cent should be treated as a finding about the instrument rather than quietly discarded. Meade and Craig (2012) found two patterns of carelessness needing different indices, and Curran (2016) recommends routine screening, so the team should report the screening procedure and the rate. Because Meade and Craig and Curran locate the problem in the data while Krosnick (1991) locates it in question difficulty, the plan should also shorten and simplify the hardest items before screening removes the cases the design provoked.

The report has confused the level at which the finding holds. Morgan (1997) and Kitzinger (1994) treat the group as the unit of analysis, and the interaction between participants as the object of study, so 'the participants agreed' is a statement about a group discussion rather than a tally of eight individual opinions. Smithson (2000) names the features that make the discussion interpretable, among them domination by some participants and tendencies towards normative discourse, and the participant who never spoke is direct evidence that the room suppressed at least one account. Barbour (2005) calls this the common error of choosing a group for convenience and then reading its output as individual opinion. The agreement should not be read as a measure of demand. Moscovici and Zavalloni (1969) found that discussion to consensus shifts attitudes towards the extremes of a scale, and that the shift persists in post-consensus individual ratings, so a group that agrees is not a sample of average opinion. Isenberg (1986) attributes the movement to social comparison and persuasive argumentation acting in combination, which means part of the agreement was produced by the discussion itself. The team cannot infer that the feature will help users; it can infer only that this group, once convened, moved towards approving it. The redesign should match the method to the evidence the decision needs. If the question is how many users would adopt the feature, the group is the wrong instrument. Guest and colleagues (2017) randomised participants to focus groups or individual interviews and found similar numbers of unique items per event, but individual interviews were better at breadth per person, and sensitive disclosures were more likely in a focus group. Because missed doses are sensitive, the group setting may actually have raised disclosure, so the sensitive material is worth keeping. A defensible design would run individual interviews for the personal accounts of missed doses and use a separate, well-composed group to observe the shared norms and vocabulary around medication. Diehl and Stroebe (1987) warn that a group does not safely contain what its members bring, since production blocking suppresses ideas, so any idea generation should use the nominal group technique of Delbecq and Van de Ven (1971), which generates ideas individually before discussion. If the team needs a numeric claim, it should run a survey or a trial rather than read one from the groups.

The proposal fails the unit's central test, because it treats a clause accepted at sign-up as consent for a study that clause never named. This is the structure of the Facebook case: Kramer, Guillory and Hancock (2014) altered the News Feed for 689,003 users and offered the platform's Data Use Policy as informed consent, and the journal's expression of concern named the absence of informed consent and the opportunity to opt out. An app's terms of service name the service, not this term-long study of low mood, and they offer no route by which a student can refuse this particular analysis while still using the app. Fiske and Hauser (2014) place the failure at the level of regulation as well as the team, since they argue that the research community must accept responsibility for studies on platform users rather than defer to a platform's terms. Grady (2015) supplies the standard the redesign should meet: consent is the mechanism through which a participant keeps a choice, so the team should offer study-specific consent that names the population, the messages, the length of the study, the withdrawal route and a cut-off after which messages already analysed cannot be removed. Fiesler and Proferes (2018) predict that relying on the terms of service will not match participant expectation. In their survey of Twitter users, 61.2 per cent of the public-account respondents did not know researchers used tweets, and 64.9 per cent believed researchers should not be able to use them without permission, with discomfort rising sharply when the whole account history was studied. Students discussing low mood are the case the survey would predict to be least comfortable, and the proposal reserves no space for a student who would refuse. The British Psychological Society (2021) makes the relevant principle explicit, since respect for autonomy and dignity, and the minimising of harm, are requirements the team must meet rather than aims it may weigh against convenience. Removing names is not anonymising, and this dataset is especially exposed because the messages are sparse, personal and longitudinal. Narayanan and Shmatikov (2008) recovered individuals from the Netflix Prize dataset of anonymous ratings of 500,000 subscribers by exploiting the rarity of certain records, and Rocher, Hendrickx and de Montjoye (2019) estimated that 99.98 per cent of Americans would be correctly re-identified in any dataset using 15 demographic attributes. Ohm (2010) explains why the legal line between identifiable and non-identifiable data cannot carry the weight the team is placing on it. The redesign should therefore keep identifiers in a separate store, report only aggregate patterns below a stated cell size, and set a deletion date, treating residual re-identification risk as a cost to be managed rather than a problem that the name-stripping has solved. Finally, the team should not assume that a review board will resolve the question for it. Vitak, Shilton and Ashktorab (2016) found that 263 online data researchers were applying the Belmont framework inconsistently, with decisions resting on individual judgement, and Metcalf and Crawford (2016) argue that the instruments of human-subjects regulation no longer describe what data researchers do. The team's duties therefore have to be built into the design, as Shilton (2018) argues, and turned into explicit procedure, as Zook and colleagues (2017) set out. A defensible version of this study would recruit students into it, offer an opt-out that preserves their use of the app, analyse at an aggregate level with a stated minimum cell size, publish its re-identification estimate before release, and state in advance what it will do with a message that discloses risk of harm.

The procedure should be stated as a sequence a reader can follow. Following Braun and Clarke (2006), the team should name the phases it actually used: familiarising with the transcripts, generating initial codes, searching for themes, reviewing them against the coded extracts and the full data set, defining and naming the themes, and writing them up, and it should say where it iterated, since the phases are not a single pass. Blandford, Furniss and Makri (2016) make the reporting demand explicit: the work behind the scenes has to be visible, or a reader cannot judge the study. So the section should show the code list, the themes that were merged, and the theme that was dropped when a counterexample appeared.\n\nThe reliability figure needs to be placed, not just quoted. O'Connor and Joffe (2020) argue that intercoder reliability can improve the systematicity and transparency of coding, and that disagreement is often more valuable than the final score, but they concede that it attests only to the coding process and guarantees neither the data collection before it nor the theme generation after it. An 85 per cent agreement figure therefore says something about the coding frame and nothing about whether the interviews asked good questions or whether the themes answer the research question. Braun and Clarke (2021) go further, arguing that quality in reflexive thematic analysis cannot be reduced to such a measure, and that the criteria that matter are conceptual clarity, engagement with the data and analytic depth. Those are the criteria the section should address directly.\n\nAn auditable trail also requires reflexivity to be documented rather than asserted. Finlay (2002) treats reflexivity as the researcher's explicit analysis of their own role and insists that its provenance and process be recorded, so the team should record what its position, recruitment and questioning may have shaped. Two coders agreeing, or one researcher returning to the data, does not remove that obligation.\n\nFinally, the claim must fit the evidence. Sandelowski (1995) argues that adequacy is a judgement about the information collected against the use to which it is put, so eight interviews can support an account of how these users experienced the booking screen but not a frequency statement about most users. The remedy is a smaller claim, supported by the quotations and the theme structure, and a stated limit on what the study can say.

By Friday I would produce a one-page brief analysis, a task analysis of how an appointment is made and kept, three scenarios with storyboards and a paper prototype of one candidate, plus notes from a critique. The brief analysis separates the goal, the constraint and the proposed solution, so the text reminder becomes a hypothesis rather than a requirement; without that separation the design work is finished before it starts. The task analysis and scenarios make the design's claim visible: which step of the patient's work it changes and which it leaves alone. Three candidates rather than one follows the parallel prototyping result, in which participants who made several prototypes before receiving critique produced better and more diverse designs, reported a larger gain in task confidence, and were far less likely to react badly to criticism. I would keep those three candidates deliberately rough. A sketch is quick, disposable and ambiguous, so its roughness invites change where a polished rendering would invite approval of the wrong concept. I would build the prototype on paper because the fidelity experiment found low and high fidelity equally good at uncovering usability issues, with results independent of medium, and because paper allows more iterations before Friday. I would test with four or five patients recruited in the waiting room, on realistic tasks, with one team member playing the computer. What I would not test is timing and visual finish: a person cannot honestly simulate response time, and a sketch cannot tell you whether the confirmation arrives while the patient is still looking at the phone. Those questions wait for a build. The limits are that the parallel study used novice designers and banner advertisements, click-through is a proxy for attention rather than quality, and one experiment is not a replication.

Both claims are weaker than the team thinks, for different reasons. The heuristic list comes from a single evaluator, and the evidence behind the method is that individual evaluators detect only between 20 and 51 per cent of the problems in an interface, while aggregating the evaluations of three to five people does rather well. So the list is a contribution to an evaluation, not a verdict, and it may also contain items that are arguable rather than wrong. Its severity ordering is a separate judgement that has not been made at all. The persona claim fails a different test. The published critique of personas is that it is impossible to determine how many, if any, users a persona represents, and that a persona cannot be verified or falsified, so its validity cannot be demonstrated; the same critique warns that a persona can be adjusted until it supports the feature a team already wanted, which is exactly the risk in a slide asserting that the persona wants one tap booking. The persona does have a defensible use, as a mnemonic anchoring a team to a specific person, provided every attribute traces to something observed. What would settle the question is a test. Give the same task to several real patients on the current design and on the one tap candidate, and compare completion, errors and where they hesitate. If the one tap booking wins for patients who fit the persona and loses for others, the decision becomes a question about who the product is for, which is a question the team can answer with evidence rather than with a slide.

A strong answer holds to one named session and describes what actually happened in it, rather than generalising about studio teaching. First locate the timing. Say who spoke, at what point in the work the judgement arrived, and whether the work could still have changed direction when it did; if the session happened after the work was submitted, the timing advantage this unit claims was not available, and the honest verdict is that the session could not have delivered it. Then apply the instruments. Schon's test is whether the exchange was a conversation with the situation, in which an action drew a reply and the next action was adjusted, rather than a performance review of a finished object; state which moment in your session answers to that description and which does not. If you use Goldschmidt and colleagues, say who dominated the talk and whether any reasoning behind the judgement was made explicit, since their coding found the teacher doing most of the talking. If you use Dannels, say which genre your session belonged to and what it trained you to do, because her analysis suggests the session teaches how to talk about work as much as how to improve it. If you use Oh and colleagues, say whether the critique matched the stage your work had reached. The second half of the answer carries the marks. Name evidence that could be collected: a recording or transcript of the session, coded for who spoke and whether a reason was given; the two or three versions of your work across the session, with the changes marked; a written note made at the time of the decision you took next. Then state a result that would count against your reading, for example that the changes you made preceded the critique and the critique only confirmed them, or that no direction changed at all. An answer that asserts the session was reflective or valuable without naming evidence of the timing, and without saying what would have shown the opposite, has not answered the question.

A strong answer holds to one named brief and works on its actual wording rather than describing briefs in general. Begin by separating three things. State the goal the brief names, the constraints it imposes, and any solution it has already assumed, since a brief that specifies a screen, a chatbot or a mobile app has chosen a solution before the problem is settled. Say which parts of the wording are agreements between stakeholders and which are unresolved, using Pohl's three dimensions to show that a requirement complete in one dimension can be empty in another. Then use the co-evolution claim. Describe a point at which your own attempt at a solution changed what you took the problem to be, or state honestly that it did not, because Dorst and Cross found the shift in nine experienced designers working to a written brief, not in every brief. If you use Alexander, show a conflict between two constraints that a list would have hidden. If you use Beyer and Holtzblatt, say what observing the real work would add to what the brief states. If you use Pacheco and colleagues, justify your choice of elicitation method from the product, the stakeholders and the information sought. The second half carries the marks. Name evidence you could collect, such as two successive problem statements from your own process with the differences marked, a note from observing the work, or a record of which constraints pulled against which. Then state a result that would count against your reading, for example that your understanding of the problem never changed across the project, or that the goal in your final work was fixed in the brief's first sentence.

A strong answer sits on one observable piece of work and produces all three artefacts rather than describing them. Begin with the task analysis. Decompose the work into goals, sub-goals and operations with plans, and mark the step your change touches and the steps it leaves alone. Use Diaper and Stanton (2004) here to say what a task model is for: a description made for a purpose, not a neutral picture, so name the purpose yours serves. Then write the scenario, naming one person, their situation and their motive, and walk them through the changed work. Use Carroll (2000) to justify keeping the claim concrete and open rather than hardening it into a requirement. Then draw the storyboard, and use Truong, Hayes and Abowd (2006) to say what the form of the drawing does to a reader's understanding, including the elements you chose for text, people, level of detail, panels and time. The argument section carries the marks: say which artefact should be the one a reader is asked to accept, and defend the choice. If you argue for the task analysis, you are claiming predictive accuracy is the standard, and you should use Gray, John and Atwood (1993), whose CPM-GOMS model predicted a 0.63 second penalty against an observed 0.65. If you argue for the scenario or storyboard, you are claiming the point is what a team can argue about, which Diaper and Stanton's purpose-built description supports. Add either Bainbridge (1983), on what automating the routine work removes from the operator, or Vicente (1999), on beginning from the constraints of the work domain rather than a prescribed sequence. Finish with the falsification condition, for example that your change did not in fact alter the step you thought it did, or that readers of the storyboard understood a different proposal from the one the task analysis described.

A strong answer produces both artefacts for one task and reports what readers actually said rather than what the student expected. Begin with the two externalisations and describe them precisely: the intentional roughness of the sketch, including what was left incomplete, out of scale or unresolved, and the specific decisions the finished rendering closed off. Use Purcell and Gero (1998) to justify expecting the ambiguous version to support reinterpretation, and Goldschmidt (1991, 1992) to explain why a sequence of sketches can reach an idea one sheet cannot, since each drawing is the input to the next rather than a record of a settled intention. The reader evidence carries most weight. Menezes and Lawson (2006) found that novice and advanced students describe the same kind of sketch differently, so a report in which the trained reader reads substantially more, or reads different things, from the rough version supports the expertise position, while a report in which both readers generate new features from the rough version but not the finished one supports the medium position. Suwa, Purcell and Gero (1998) give the vocabulary for coding what was read off, physical, perceptual, functional and conceptual, and the student should use those four levels rather than impressions. Then take a position. If the cognitive work is in the medium, cite Goldschmidt's account of the sketch as a setting in which thoughts are constructed on the fly; if it is in the designer, cite Cross (2004) on proactive problem framing and Goldschmidt (2014) on sketching being near costless only in the hands of an experienced sketcher. The falsification condition matters: for example, if neither reader produced anything the designer did not intend, or if the trained reader produced more from the finished rendering, the medium account is weakened. Close by specifying that the study the unit says is missing would hold the task constant and vary representation ambiguity and designer expertise factorially, measuring both the number of unintended features read off and the quality of the resulting design.

A strong answer runs a real exercise and reports it, then argues a position rather than listing sources. Begin with the artefacts: the five or more candidates, what distinguished them structurally rather than decoratively, and which one was carried to a prototype. Then describe the deferred critique precisely, including what it changed and what an immediate critique would have addressed instead; Osborn (1953) supplies the doctrine of deferred judgement that justifies the arrangement, and Dow, Fortuna and colleagues (2011) support expecting a deferred, multi-candidate round to improve exploration, the quality of critique and the team's rapport. The argument then has to do real work, because the session cannot separate the two possible causes. Having several candidates is one change, and postponing the critique is another, and a single session varies them together. Dow, Glassco and colleagues (2010) have the same problem: their parallel condition made several prototypes before feedback arrived, so plurality and timing moved together. Jansson and Smith (1991) and Smith, Ward and Schumacher (1993) provide the reason to think plurality matters, since exposure to a single idea or example narrows what follows and a warning does not remove the effect, but they demonstrate fixation rather than the benefit of many candidates. Sio, Kotovsky and Cagan (2015) complicate the mechanism further by finding that examples can raise novelty and quality while narrowing variety, so more candidates is not automatically the same as better candidates. A good answer therefore takes a side and then states its falsification condition: if the deferred round changed nothing that an immediate critique would have missed, the timing account is weakened, and if having five candidates made no difference to what survived, the plurality account is. Close by naming the missing study as a factorial manipulation of the number of candidates and the timing of feedback, held constant in task and participants, with both a fixation measure and a finished-design outcome, and by acknowledging that no source in this unit has run it.

Report the exercise before arguing about it. Give the prototype, the three participants, the tasks in the words you used and the problems you found, then separate the problems about whether the interaction worked at all from those about how quickly or pleasantly it worked. That separation is what the literature supports. Walker, Takayama and Landay (2002) found low- and high-fidelity prototypes equally good at uncovering usability issues, with results independent of paper or computer medium, and Virzi, Sokolov and Karis (1996) found the same problem sets with a correlated detection proportion, so findings about the presence of a problem are the ones a paper test earns. Lim, Pangam and colleagues (2006) found prototype characteristics shaping which problems appeared, so the problems your medium made visible are a subset rather than the whole set. Sauer and Sonderegger (2009) found that a computer prototype can overstate completion time and that users overrate the aesthetics of reduced-fidelity designs, and Sauer, Franke and Ruttinger (2008) found paper prototypes may overestimate a design change's behavioural effect. A strong answer distrusts its own efficiency and affect findings, says so, and names the dimension to change in Houde and Hill's (1997) terms rather than proposing to raise the fidelity overall.

A strong answer produces an actual persona, not a description of one: a named character, the observations each detail comes from, and a note of which details are inferred rather than observed. It then separates what the artefact can support from what it cannot. On the supporting side, Pruitt and Grudin (2003) argue from three years of use that personas engage a team more than scenarios alone, convey qualitative and quantitative data, and focus attention on what other methods miss, and Salminen, Jung, Chowdhury, Sengun and Jansen (2020) provide measured evidence that a persona-based presentation of data gave faster task completion and higher identification accuracy than an analytics system on identical data. Nielsen (2019) supplies the condition that makes this defensible: a persona grounded in data and used as a shared frame rather than a portrait of a user. On the unsupporting side, Chapman and Milham (2006) argue that it is difficult to determine how many users a persona represents, that it cannot be verified or falsified and so has no demonstrable validity, and that it can undermine a team's ability to settle questions with data. Matthews, Judge and Whittaker (2012) add that practitioners used personas almost exclusively for communication rather than design, and judged them abstract, impersonal, misleading and distracting. The honest conclusion is that the persona may sit in the requirements document as a shared reference and a summary of what was observed, marked as such and traceable to the interviews, but it cannot be presented as a finding about the patient population, because nothing in it states what observation would show it to be wrong. The evidence that would change this answer is a controlled study in which personas built from documented data drive specific design decisions and the resulting designs are compared against a control team given the same data without personas.

A strong answer draws two artefacts and keeps them distinct. The blueprint runs the appointment from booking through reminder, arrival and consultation, marking beneath each step the on-stage contact, the back-stage work and the systems the service depends on. The journey map follows one named patient through the same sequence and marks where the experience drops. The argument then separates three claims. First, the blueprint can make a dependency visible. If the reminder leaves a system the reception team does not control, and the patient's reply reaches a staffed phone only at certain hours, the blueprint shows that the failure sits behind the interface. Zeithaml, Parasuraman and Berry (1985) supply the reason to look there, since their gaps model locates failure in the distance between what customers expect, what management perceives and what is delivered. Second, the blueprint cannot show that the fix works. Shostack (1982) makes the service designable and Bitner, Ostrom and Morgan (2008) claim a blueprint exposes failure points, but their warrant is a lineage of worked cases that measures no outcome, so the blueprint is an argument rather than a finding. Third, the map is only as good as the observations behind it. Rosenbaum, Otalora and Ramirez (2017) argue that many maps are unrealistic because their authors filled the gaps with assumption, and Folstad and Kvale (2018) find the journey concept inconsistently defined, so every run of the map should trace to an interview or an observed encounter. The strongest objection is Wetter-Edman and colleagues' (2014), that value is co-created, so an apparent failure point may be where the patient's own situation diverged from the clinic's model rather than a defect the clinic can repair. The evidence that would change this answer is a study that builds the blueprint from documented observations, states in advance what would show it wrong, and compares outcomes for a clinic designing from the blueprint against one given the same observations without it. Trischler and colleagues (2018) come closest, since customer involvement in service design teams improved the novelty and usefulness of the ideas produced, but that measures ideas rather than missed appointments.

A strong answer proposes a shallow, broad top-level structure built from the students' own groupings and then defends each part against a named source. It draws the first level from the card sort, since Faiks and Hyland (2000) found that users' categories differed from the designers' assumptions, and it states plainly that a card sort shows agreement, not correctness, so it cannot settle which grouping is best. It justifies minimising depth with menus of eight or nine selections on Kiger (1984) and a broad first level with an index on Snowberry, Parkinson and Sisson (1983), and it handles the rule of seven carefully: Miller (1956) is about immediate memory and reports no experiment on interfaces, so the number is used only as weak support and never as a constraint. It treats the labels as part of the architecture, citing Larson and Czerwinski (1998), because the match between link text and the reader's goal affected retrieval as much as the grouping did, and it uses the search logs as evidence of the words students actually use, which is a scent signal on which Chi and colleagues (2001) and Fu and Pirolli (2007) show navigation can be predicted. It names the strongest objection honestly: the strongest counter-argument is that these findings come from menu-search experiments with artificial targets and from models fitted to web logs, so they support a proposal but do not prove a working portal better; Kiger's depth and breadth advantage itself interacted with the task, so no shape is best for every search. The study that would settle it crosses the proposal against the alternative in a factorial design that varies structure and label quality, uses pre-registered time-to-find and error measures on realistic tasks, tests students who know the portal and students who do not, and reports a named threshold of evidence that would reverse the recommendation.

A strong answer refuses to choose on the basis of the longer list and treats the two reports as different nets. It notes that a single evaluator finds only part of the problems while aggregating three to five evaluators does rather well (Nielsen and Molich, 1990), and that the yield of inspection follows a curve whose best ratio of benefit to cost comes at about four evaluations (Nielsen and Landauer, 1993). It then reports that testing was found to identify the largest number of problems, including severe ones the walkthroughs missed, while about a third of significant problems were common to every method (Karat et al., 1992), and that heuristic evaluation was best for serious problems at low cost but also produced many low-priority findings (Jeffries et al., 1991). Against any claim that either list is authoritative it sets the evaluator effect: evaluators using the same method on the same interface detect markedly different problems, with average agreement between two evaluators of 5 to 65 per cent (Hertzum and Jacobsen, 2001), and the professional teams in the comparative series found that most problems were reported by one team only (Molich et al., 2004; Molich and Dumas, 2008). The recommendation is to keep both sets of findings and to write, for each problem, the evidence behind it, the option it rules out and the criterion it fails, in the manner of questions, options and criteria (MacLean et al., 1991), keeping that reasoning as a navigable artefact (Conklin and Begeman, 1988) rather than reducing the decision to a score. The strongest objection is that each evaluation is a single small sample, so neither set of findings is a population estimate of what users will meet. A study that crossed method with evaluator, pre-registered a problem list and reported detection and false-alarm rates for several teams per method would settle the question, and a finding that the two families detected the same severe problems at the same rate would change the recommendation.

A strong answer measures before it recommends. Begin by naming the chain for one real interaction: the domain lookup, the connection setup, the secure handshake, the request and its response, and any subresources the first screen needs. A browser performance panel, or a trace of the request, gives a figure for each stage, and the sum should match the delay the person perceives. Compare the total against a stated target, such as the tenth of a second that reads as instantaneous or the one-second limit that preserves the flow of thought. If the chain exceeds the target, locate which stage dominates. A distant origin points towards a content delivery network; repeated uncached resources point towards caching; several sequential round trips point towards issuing requests in parallel or combining them; a slow response that arrives after a fast network points away from the network entirely, which is a common misdiagnosis. The recommendation should name the single change with the largest measured effect, state the expected saving in milliseconds, and say how it would be verified, for example by re-measuring the same chain. A recommendation that reports a total but cannot say which part produced it is not yet a measurement.

A threat model argues backwards from the opponent to the control. A strong answer names one consumer service, states who would attack it, what they stand to gain, and what they would do, and only then asks what the user is asked to do about it. For a second-hand marketplace, the opponent might be a fraudster seeking account takeover rather than an eavesdropper on the wire, which shifts the priority away from transport encryption, which is already standard, and towards session theft, phishing and weak account-recovery flows. Encryption and hashing should be placed correctly: a deliberately slow salted hash for stored passwords, encryption for data in transit and at rest, and no reuse of one secret for two purposes. Then the critique. Following Herley, if a control depends on the user reading and acting on a prompt that almost everyone dismisses, the burden falls on the many to avert a loss that falls on the few, and the rational response for each individual is to ignore it. The better design removes the decision or makes the safe path the default, using fail-safe defaults and least privilege rather than more user education. The answer earns its marks by naming the evidence for each claim and by admitting where that evidence is thin.

The engineer's proposal assumes that delay is caused by computation, and that a faster computation will show the screen sooner. The sources do not support that as the main account. Wilkes (1965) proposed a fast store holding recently used blocks precisely because programs reuse a small set of items, and Denning (1967) made that reuse measurable as the working set, so the question is which tier holds the items the first screen needs, not how many instructions decorate them. Wulf and McKee (1995) projected that processor speed would outgrow memory bandwidth, and Patterson (2006) showed across memory, disk and network that bandwidth improves far faster than latency, which predicts that cutting arithmetic will not remove a fetch from a slow tier. The strongest case for the rewrite is that decoration that touches many items can itself force fetches: if the old code walks a large structure or re-reads each item from a colder tier, then reducing that work reduces movement, and Smith (1982) shows the top tier's mapping, replacement and write parameters decide whether reused items stay close. The strongest case for placement is the production evidence that the tier can be chosen: Lagar-Cavilla and colleagues (2019) place cold pages in a far tier, Maruf and colleagues (2023) promote and demote pages between a fast and a slow tier with less than one per cent gap from all-local, and Li and colleagues (2023) pool memory while staying within 1 to 5 per cent of all-local performance; Ousterhout and colleagues (2010) push the limit by keeping storage in DRAM, and Mutlu and colleagues (2022) show the opposite move of taking computation to the data. The argument also has to pass the storage tier, since Chen and colleagues (1994) show it is engineered for reliability as well as speed and Agrawal and colleagues (2008) show flash performance depends on access pattern, not raw bandwidth, so a grid read at startup is a placement question before it is an arithmetic one. Gray and Shenoy (2000) give the form of the test: compare the cost of fetching the first-screen items against the cost of holding them in the tier above. The measurement that would settle the dispute is a controlled trace of the first-screen load under both builds, reporting time spent waiting on each tier rather than instructions executed, with the device and the network condition held fixed. The observation that would count against the rewrite is a first-screen latency that does not move while the per-tier wait times show the delay sitting in the fetch: that would put the cost in placement, not in arithmetic. Schroeder and colleagues (2006) make the condition explicit, because a benchmark that closes the system can mislead about latency, so the trace must name its workload model and its network state to be admissible. If instead the per-tier waits fall with the new code, the engineer's account survives, and the honest result is that the rewrite moved data as well as work.

The engineer's proposal assumes that the freeze is caused by computation, so a faster computation appears sooner and the freeze shortens. The sources do not support that as the main account. Liu, Xu and Cheung (2014) set out Android's single thread policy: one main thread, also called the UI thread, dispatches system calls and user events, so lifecycle handlers and interface event handlers all run there. Their study of 70 fixed performance bugs from eight widely used applications found 53 to be interface lagging, and the largest of three recurring causes was lengthy operations in main threads. Their example was repaired by moving the work to a background thread, which changes who does the work rather than how fast it runs. Corbato, Merwin-Daggett and Daley (1962) supply the older form of the same reasoning: a shared machine feels prompt because the processor is reallocated, not accelerated, and a scheduler can switch between threads but cannot make a thread finish its work. The strongest case for the rewrite is that parsing on the interface thread often forces fetches and decoding work, so cutting that work reduces the time the thread is occupied, and Liu and colleagues' numbers allow for that: 11 of 52 attributable bugs were lengthy main-thread operations, so the class is real even if it is not the only one. The strongest case for scheduling is measured. Sandhi (2025), across 47 million Android devices, attributes the largest root cause to applications hogging resources under optimistic process management, and reports that remodelling process management cut slow responsiveness by 60 per cent and battery use by 10.7 per cent, a change made below the application. If the fix is to move the parse off the interface thread, the price is shared state: Lampson and Redell (1980) document the difficulties monitors meet in a real system, and Lu, Park, Seo and Zhou (2008) found that most of 105 real-world concurrency bugs were not fixed by adding or changing locks. The measurement that would decide the question is a trace of the first screen's opening on a fixed device and build, reporting the interval during which the interface thread was occupied and the tier each blocking operation waited on, with the engineers' rewrite and the off-thread version compared under the same workload. The observation that would count against the team's preferred fix is a reduction in the parse time that leaves the total freeze unchanged because the thread is still occupied, whether by the parse or by the fetch and decode that follow it. The answer should also name the owner of the decision, using one of the remaining sources: Boehm (2005) to rule out a library-only fix, Anderson and colleagues (1992) for the kernel boundary, Lee (2006) for language-level concurrency in which non-determinism is explicit, or Tu and colleagues (2019), whose 171 Go concurrency bugs show that changing the mechanism relocates the fault rather than removing it.

The team's proposal assumes that processing is the dominant term, so a faster handler shortens the whole sum. The sources do not support that as the default reading. The screen is reported as usable only after 1.4 seconds, which is above the 1 second limit Nielsen (1993) associates with an uninterrupted flow of thought, so the transaction is mis-budgeted by a fixed threshold rather than measured: Miller (1968) asks response time to what, and Galletta and colleagues (2004) show the cost of delay is a curve whose shape depends on the task, flattening at about 4 and 8 seconds. Little (1961) gives the reason a queue matters most: at a fixed arrival rate, the average time a unit spends in the system rises with the average number in the system, so a queue allowed to stay full is delay that a faster handler does not remove. Gettys (2011) describes the condition in which this dominates, bufferbloat and bad latency under load, and Nichols and Jacobson (2012) give the mechanism and the measurement, edge queueing delays from hundreds of milliseconds up to multiple seconds arising when the sender's window exceeds what the path can hold. The strongest case for the rewrite is that processing is a real term and the microbenchmark may be honest: if the handler blocks on work the interface thread also needs, cutting that work shortens the occupancy of the thread, and the fix is a genuine reduction in one term. The strongest case against is structural. The 1.4 seconds may be a fan-out, and Dean and Barroso (2013) show the slowest of many parallel parts sets what the user sees, so no single handler halves the total. The time may be round trips: Mathis and colleagues (1997) predict a connection's bandwidth from its loss rate and round-trip time, so a fixed payload takes longer over a longer path, and the control law is chosen by other endpoints under CUBIC (Xu and colleagues, 2023) or the alternatives BBR explores (Cardwell and colleagues, 2016). The measurement that would decide it is a trace of one first request on a fixed build and device, reporting the four terms separately: the round-trip propagation estimate, the transmission time for the payload, the server and client processing time, and the time packets spent queued, together with how many round trips the transaction required. One observation would count against the team's account: if the sum of queueing and extra round trips exceeds the processing time, then halving the handler leaves the screen slower than the budget the sources imply. If queueing dominates, the repair is at the queue, and FQ-CoDel (Hoeiland-Joergensen and colleagues, 2018) shows the queue is controllable, while De Schepper, Bagnulo and White (2023) warn that the root cause may sit in the senders' controllers instead.

The proposal addresses one exposure and one failure. It hides the lookup from an observer between the client and the resolver, which the transport routes are built to do, and it moves resolution off a resolver the network may not control, which matters because Schomp and colleagues (2013) found that only nineteen per cent of open resolvers consistently returned correct time to live values and that twenty per cent imposed at least one hundred milliseconds of delay before a query left the resolving infrastructure. It does not address the failure that has nothing to do with encryption. Wang (2015) shows that a forged answer placed in a resolver's cache is reused until its lifetime expires, which is a property of caching that Mockapetris (1987a, 1987b) builds in, and no transport encryption prevents it. It does not address integrity either, since Arends and colleagues (2005) state that the security extensions, which do provide origin authentication and integrity, explicitly do not provide confidentiality, so the proposal buys the opposite of what a signature buys. The strongest case for the proposal is real. Plain resolution is readable on the path, the encrypted alternatives are standardised and affordable, and Zhu and colleagues (2015) measured roughly nine per cent additional latency to the recursive resolver, so the friction is small. Bortzmeyer, Dolmans and Hoffman (2021) can reduce what each stop in the lookup learns even against an observer. The strongest case against is that it concentrates observation rather than removing it. Siby and colleagues (2020) place an adversary between client and resolver and still identify visited domains using one hundred and twenty-four times less data than attacks on HTTPS flows, finding the standardised padding ineffective, so the provider the hospital trusts becomes the party with the clearest view. ververis and colleagues (2025) show where that concentration lands: more than ninety-seven per cent of enabled resolvers delegated to major cloud providers, and configuration errors would leave a compliant client failing to upgrade in over ninety-nine per cent of observed cases. Lu and colleagues (2019) add that availability is not use, since few clients had adopted what was already reachable. The measurement that would settle the choice is an inference experiment run on traffic padded at the overheads the deployment actually uses, with the adversary position stated in advance, comparing what one provider can infer across its customers against what an on-path observer can infer with no encryption. The observation that would count against the proposal is a single provider identifying the visited domains from the padded traces of its own clients. Chung and colleagues (2017) give the reason to expect a deployment to fail on its own, since adoption of a naming security property waits on intermediaries rather than on the site that wants it. Partridge, Mendez and Milliken (1993) supply the alternative that needs no change of resolver at all, since an anycast address lets the routing system move a request away from an instance that has failed.

The proposal amortises one cost and moves another. It amortises the TLS handshake, which Dierks and Rescorla (2008) fix at two round trips in TLS 1.2 and Rescorla (2018) reduces to one, or none on resumption, and it amortises connection setup through HTTP/2 multiplexing (Belshe and Peon, 2015) carried onto QUIC and HTTP/3 (Iyengar and Thomson, 2021; Bishop, 2022). It does not remove statelessness. Because the protocol remembers nothing (Fielding, Nottingham and Reschke, 2022), the session state still has to live somewhere, so the proposal moves memory of the visitor into a resumed session and a cache rather than into the request. The strongest case for the proposal is that the handshake toll is real and repeated, so paying it once per session is a genuine saving, and Langley and colleagues (2017) report measured gains from exactly this kind of reuse at internet scale. The strongest case against is that the saving is conditional. Varvello and colleagues (2016) found most sites advertising HTTP/2 did not serve over it, Trevisan and colleagues (2021) found the HTTP/3 gain dependent on network conditions, and Ruth and colleagues (2018) found the protocol concentrated in a few providers, so a long-lived connection helps only where the path and the intermediaries cooperate. Enabling a post-quantum key share raises the handshake cost it is trying to amortise: Google (2024) measured a Kyber share at roughly 1 kilobyte against 32 bytes, adding about 4 per cent to the median handshake time on desktop and more on mobile, and Sikeridis and colleagues (2020) measured handshake latency increases of 1 to 300 per cent depending on the algorithms. Aas and colleagues (2019) add the caution that the binding cost of encryption was the trust setup, not the handshake. The deciding measurement is a field experiment that verifies the version actually negotiated, holds content constant and reports the distribution of user-visible timing across diverse networks for resumed against fresh connections. One observation that would count against the proposal is a population in which resumed long-lived sessions show no improvement in user-visible timing over fresh connections.

The fixed ten-minute lifetime buys latency and moves the staleness cost rather than removing it. A cache pays only because requests repeat, and Breslau and colleagues (1999) show a heavy-tailed, slowly changing popularity, so a small edge cache will serve most of the homepage traffic quickly. What the proposal moves is the invalidation decision: instead of the origin deciding freshness per response, a single fixed lifetime holds every page equally, so the newsroom's correction and the replacement of a lead story arrive up to ten minutes late. Fielding, Nottingham and Reschke (2022) place the freshness lifetime with the origin, and Nottingham (2010) offers the better instrument, since stale-while-revalidate would let the edge serve the old page immediately while revalidating, and stale-if-error would keep the site readable when the origin fails. The strongest case for the proposal is that placement itself lowers latency without any change to content: Nygren, Sitaraman and Sun (2010) map each client to a measured-nearest server, and Goel, Wittie and Steiner (2015) cut page load time by 29 per cent on average by letting the client help choose, so the edge is where the gain is. The strongest case against is that proximity does not deliver what it promises unless the cache is kept coherent and correctly chosen: Krishnan and colleagues (2009) found that least-latency selection did not suffice because queueing delays overrode the benefit of a nearby server, and Nishtala and colleagues (2013) show that consistency costs explicit machinery, since the cache may lose data without notice and invalidation is not transactional. A fixed lifetime also ignores Rizzo and Vicisano (2000), who found that the replacement policy, not capacity alone, sets a cache's benefit, so a uniform ten minutes would evict breaking news as eagerly as an archive page. The deciding measurement is a controlled comparison, over one news cycle and one client population, of no cache, an origin-controlled edge cache with per-page directives, and a fixed ten-minute edge cache, reporting together the share of responses served stale, the fraction of requests that observe a correction after it was published, and the user-visible latency distribution. An observation that would count against the fixed proposal is a measurable share of correction views during the stale window that a per-page directive would not have produced.

The proposed policy buys measured strength and moves the remaining cost to the service rather than removing it. Komanduri and colleagues (2011) found that a 16-character minimum produced significantly less predictable passwords than an eight-character policy with character-class rules while being less onerous for users, and Ur and colleagues (2015) show that composition rules are satisfied with predictable edits, such as a trailing exclamation mark, so the rules being dropped were buying less than they appeared to. Screening attacks the reuse that the measurement found: Florencio and Herley (2007) recorded 6.5 passwords each shared across 3.9 sites, and Pal and colleagues (2019) compromised more than 16 per cent of accounts in fewer than a thousand guesses once another password was known. Temoshok and colleagues (2025) make each of these a requirement, and Biryukov and colleagues (2016) make the storage cost a parameter the service sets. The strongest case for the proposal is that it moves work from the user to the service and closes the attack the literature actually measures. The strongest case against is that the burden does not disappear: Pal and colleagues (2019) place the effective defence in a personalised meter on the user's own device, Bonneau and colleagues (2012) found every replacement scheme worse than passwords on deployability, and Silver and colleagues (2014) show the packaged substitute for memory failing at its autofill interface. A longer secret and a screening list still ask a person to recall a credential and to answer a rejection prompt. The measurement that would decide it is a before-and-after comparison within the same user population, reporting guessed-password rates and account-takeover rates together with reset volume, creation time and abandonment. If takeover rates fall while resets and abandonment rise, the policy has moved the cost rather than removed it.

Face recognition used as the sole factor redirects the attacker rather than removing the route. Pankanti and colleagues (2002) show a match is a probabilistic claim about noisy measurements, and Matsumoto and colleagues (2002) show a trait can be presented rather than held, so the attacker moves from guessing a password to presenting a copy of a face, with no held secret to change. Jain and colleagues (2016) conclude that a multi-factor combination with a password or token may be the better route, because traits lack secrecy and can link a person across applications. The strongest case for the proposal is operational: a camera asks nothing of a busy clinician, and Holtgrave and colleagues (2025) show that the obstacle to authentication deployment is organisational rather than technological, so removing a token can succeed where a second factor failed. The strongest case against is the measured unevenness of error. Buolamwini and Gebru (2018) found error rates up to 34.7 per cent for darker-skinned women against 0.8 per cent for lighter-skinned men, and Grother and colleagues (2019) found false-positive rates varying across demographics by factors of ten to beyond a hundred; a false accept of that size in a clinical setting admits the wrong person to a record. Terhorst and colleagues (2022), Fu and Damer (2022) and Kotwal and Marcel (2024) disagree about whether the cause is acquisition, model or dataset, but none of them shows the gap closed. The measurement that would decide the question is a within-hospital trial that runs face, token and combined authentication on the same staff and reports per-group false-accept and false-reject rates together with the time and help-desk cost of each. An observation that would count against the proposal is a false-reject rate high enough to push clinicians into shared or borrowed accounts, which would trade a redirected attack for a new one.

The assessment must begin by naming the adversary and the assets, because Anderson (2020) treats the threat model as prior to any mechanism: a control sized for a vandal is absurd against a state, and a control sized for a state is unaffordable against a vandal. Myagmar and colleagues (2005) supply the failure this avoids, since security requirements are usually written before any adversary is named, so the assumptions behind a security claim stay unstated. The assets are the customer records, but also the trust the service places in its dependencies: Ohm and colleagues (2020) analysed 174 malicious packages distributed through npm, PyPI and RubyGems, so libraries the team never chose are part of the asset list. Saltzer and Schroeder (1975) give the requirements their shape through eight principles, and Yoder and Barcalow (1997) show the principles as structure, such as a single access point and check points. The strongest case for a formal or checklist method is that it makes the sizing comparable and the coverage repeatable: Manadhata and Wing (2011) formalise the attack surface and validate the metric through three exploratory empirical studies, so exposure can be compared between versions, and Schneier's (1999) attack trees compare defences against the cheapest path rather than the most likely attack. The strongest case against treating such a method as sufficient is twofold. First, the boundary it draws leaks by construction: Lampson (1973) shows a channel can be built from almost any shared resource, including timing, so absolute confinement is impractical and the goal is to bound the leak. Second, the evidence that the canonical methods improve security is thin: Xiong and Lagerstrom (2019) assessed 176 articles and selected 54, finding the domain lacks common ground and that most threat modelling remains manual with limited assurance of validation. Harrison, Ruzzo and Ullman (1976) add that no general algorithm decides whether a subject can acquire a given right, so completeness cannot be proved in general, and the cases of Spafford (1989) and Langner (2011) show models failing at the level of trust and of the wrong asset rather than of code. Souppaya and colleagues (2022) show the institutional pressure to adopt the framework regardless. The measurement that would settle the question is a comparison of two or more named methods on the same service with the same team, following the system far enough to count vulnerabilities found, fixed before release and exploited later, together with the attacks the model did not anticipate. An observation that would count against the adopted method is a later incident that travelled through a dependency the threat model never listed, since that would show the asset list, not the checklist, was the point of failure.

The assessment must start by naming the decision, because the unit's claim is that the control, not the user, is the tractable fault. The drafted flow asks the student to judge whether a prompt is genuine, to keep a second device available every fourteen days, and to interpret a page of explanation, and Stanton and colleagues (2016) show why that accumulation is the problem: their interviews with 40 users found more than half alluding to security fatigue, expressed as resignation, loss of control, fatalism, risk minimisation and the avoidance of decisions, produced by many demands rather than one. Anderson (2001) supplies the structural reason the cost lands on the student, since the party who bears the cost of a failure is often not the party who chooses the protection. The strongest case for the security officer is empirical and should be stated fairly: Kumaraguru and colleagues (2010) evaluated embedded phishing training, which delivers the lesson when a user clicks a simulated link, and found the trained group's false negative rate fell from 0.40 to 0.11, with the knowledge retained over a week and longer in a field deployment and no rise in responses to legitimate mail, so instruction demonstrably changes behaviour and the failure might indeed be the student's. The strongest case for the designer is that this control, unlike a phishing email, offers no decision the student can evaluate: Felt and colleagues (2012) found 17 per cent of participants attending to permissions at installation and only 3 per cent answering all three comprehension questions correctly, with the screen asking for a decision the user could not act on. The same asymmetry appears in the warning record, where Sunshine and colleagues (2009) found that every warning strategy they tested, including two they designed, left too many participants vulnerable and recommended blocking unsafe connections instead of warning about them, and Akhawe and Felt (2013) found clickthrough ranging from 9 to 23 per cent for malware and phishing warnings to 70.2 per cent for Chrome's secure-connection warnings. Whitten and Tygar (1999) had already shown the shape of the problem, since only about one third of twelve experienced users could sign and encrypt a message with PGP 5.0 in 90 minutes even though the tool's cryptography was sound. The measurement that would settle it is a field study that enrols students into two flows on the same service, one that prompts and explains and one that sets the safe configuration by default, and follows each group to a measured outcome: completion of enrolment, support contacts, successful account recovery and, crucially, the harm that the second factor was meant to prevent. Herley (2016) requires that last term, because a claim that the default is better is a claim of improvement that no observation can refute unless an outcome is measured, and Franz and colleagues (2021), whose taxonomy of user-facing phishing interventions found experimental setups often lacking in realism and user effort scarcely considered, point to the same gap. One observation would count against the default: if the defaulted group showed more account compromises than the prompted group, because students who never chose the second factor treated it as optional and disabled it later, then removing the decision would have removed the wrong thing, and the honest response would be to measure the burden rather than assume it.

The description should stay neutral and specific: the checkout completes the order when the primary button is pressed, and there is no step that asks the buyer to confirm before payment is taken. The interpretation is the claim about what the design does. It treats commitment as an obstacle to be removed, and it places the cost of error on the buyer rather than on the flow. The judgement needs a stated criterion, and the strongest one here is reversible against irreversible action. That criterion supports the judgement that the design is wrong, because it makes the one action that cannot be undone the action that is easiest to take by accident. The evidence objection is separate from any preference. The team asserts that friction was removed, and that assertion has not been tested. No measurement shows whether accidental orders rose after the change, and none shows a fall in abandonment large enough to justify the risk. A preference objection would be that the critic simply likes confirmation screens, and that objection is weak because it cannot be met by data. The evidence objection names the study that would settle it. Compare accidental order and refund rates before and after removing the step, at the same traffic and with the same mix of buyers. If accidental orders are unchanged, the friction claim survives. If they rise, the design has moved a cost onto the buyer, and it should be reversed or given a cheap undo. The purpose of the critique is to leave the team a testable next action.

A defensible answer commits to one model and argues from the situation rather than from preference. A council redesigning a live service usually has a defined problem, real users with concrete stakes, and a delivery date, so participatory design is the right place to start. Its commitments are the reason. It treats users as full participants rather than subjects, it shares the framing of the problem, and it aims to strengthen the party with least power over the outcome, which is the position of a resident who depends on a service they did not choose. The Double Diamond would supply a shared language for divergence and convergence and little more, because it does not say whom to involve or how to weigh what they say. Speculative design would be a poor first move, because it produces questions about futures rather than a specification, and the service has to run on Monday. The answer should then name the cost of the choice. Participatory design makes a project slower and less predictable, because participants can reframe the problem after the timetable is agreed, and their criteria are not the council's. It also demands that the council hand over real influence, or the activity degrades into consultation with better paperwork. A strong answer adds the safeguard that follows from that risk. Agree in advance which decisions participants can change, and say so honestly, so the promise made at the first workshop matches the authority actually available.

A strong answer names the model, names one decision, and gives the evidence that the decision would have gone the other way without the model. Begin with March and Smith (1995), who separate four kinds of research output and place a method among them, so a process model is an artefact assessed by what it does rather than by whether it describes nature. Use the Design Council (n.d.) to state what the model actually claims: four phases, Discover, Define, Develop and Deliver, and the Council's own description of the diagram as holding irrespective of the methods and tools used. That description is the difficulty the answer has to face, because a device that holds for any project cannot by itself supply the inferential step from an observation to a decision. Use Wynn and Clarkson (2018) to say what the model is for, since their framework treats each process model as a representation made for a purpose, and then name the purpose this model served on this project. Use Dorst and Cross (2001), whose nine experienced designers co-evolved the problem and the solution, to test whether the decision you claim was a phase decision or a reformulation of the problem. If you argue the model changed no decision, say so plainly and use Simon (1973) to argue that the apparent difficulty lay in the solver's memory and search rather than in the problem. If you argue that it did change a decision, use Badke-Schaub and colleagues (2007) and claim only what their evidence supports, which is that a shared frame may be the mechanism rather than a constraint on inference. The closing paragraph states the evidence that would count against the claim, such as a record of the option that was live before the phase and the option chosen after it.

A strong memorandum separates the three claims before pricing any of them. Begin with Kimbell (2011), who distinguishes design thinking as a cognitive style, as a general theory of design, and as a resource for organisations, and who argues that the first two rest on a dualism between thinking and acting and that all three privilege the designer as the main agent. Say plainly that the firm is buying the third, and that the first two cannot be bought. Then state what the third has to show. Use Nussbaum (2011) to name the risk precisely: he argues that companies welcomed design thinking because it arrived as a process, and that firms turned it into a linear, gated methodology whose best result was incremental change. That is the failure mode to design against, so require the programme to specify what decisions it will change rather than which workshops it will run. State the strongest case for proceeding, which is Nakata and Hwang (2020), whose survey of innovation managers found that design thinking comprises six interrelated mindsets and actions, that it strengthens new product and service performance, and that the effect is robust across levels of market turbulence. Do not dismiss it. Instead set the demand that follows from Micheli and colleagues (2019), who find promoters and critics disagreeing about attributes, applicability and outcomes and warn that the construct is threatened by collapse, so require the firm to fix the definition it is buying and the outcome it expects before the programme starts. Add the fidelity condition from Carlgren and colleagues (2016), whose five themes, drawn from six organisations, show a gap between the idea of the method and its enactment, so two teams reporting that they use design thinking may be doing different things. Require the programme to record what each team actually does. Add the phase condition from Seidel and Fixson (2013), whose fourteen novice teams showed methods helping in both phases while more brainstorming sessions corresponded to lower performance and reflexivity helped generation but hindered selection, because a single aggregate measure would hide those opposing movements. Close by naming the observation that would end your support: if the teams cannot produce a decision that went differently because of the programme, and a comparison team without it reaches the same decisions, the money bought the mood rather than the method.

Begin by separating what the label adds from what it hides. The label adds a warning that the unit endorses: as Ritchey (2013) reproduces them, Rittel and Webber's criteria include that a wicked problem has no definite formulation and no stopping rule, so the authority should not expect to close the problem statement before work begins, and it should not treat an early phase of understanding as a precondition for proposing anything. Then state what the label leaves undetermined. Head (2008) reduces wickedness to high levels of complexity, uncertainty and divergence, which is a definition rather than a diagnosis, and it does not tell the authority what to do on any particular Tuesday. Peters (2017) presses the point that many problems labelled wicked fail a strict definition of it, so the word may be doing rhetorical work; a briefing should ask which of the three characteristics is actually high, and for whom. Turnbull and Hoppe (2019) supply the alternative reading, that wickedness is better understood as the political distance between those who question the problem, which is high only when wide distances are maintained, and that has a direct consequence for the programme: the useful work is to find sub-questions that can be closed while the wider disagreement stays open. Farrell and Hooker (2013) then discipline the diagnosis, since the ten features derive from agent finitude, system complexity and problem normativity, which science and design share, so the authority should grade the challenge rather than decide whether the problem is wicked or tame. van Bueren and colleagues (2003) show why the grading must include the network: in their environmental case, impasses and breakthroughs were explained by cognitive, social and institutional factors, and deadlock and progress turned on the interactions of the network rather than on properties of the problem. What I would measure follows. I would ask for the divergence: which stakeholders disagree, about what, and whether the disagreement is about facts or about aims. I would ask for the network: who can block, who can deliver, and what has changed in that set. And I would ask the frames: Cross (2004) reports that expert designers move rapidly to early solution conjectures and use them to define the problem, so I would ask the team to produce two or three candidate frames, following the frame creation practice Dorst (2015) sets out as a nine-step approach to open, complex, dynamic and networked problems, and to show what each frame makes visible and what it hides. The observation that would change my advice is a low divergence score combined with a stable network, since that would mean the problem is being called wicked when its difficulty lies in delivery, and the authority would be better served by a plain programme with a fixed outcome measure.

Begin by naming the assumption. The plan assumes that fidelity is a single axis of progress, so more fidelity yields more knowledge, and that one prototype can answer whatever the team needs to know. State what the unit says instead. Houde and Hill (1997) argue that a prototype is defined by the question it answers, not by its finish: a role prototype asks whether the artefact is useful, a look-and-feel prototype asks what using it is like, and an implementation prototype asks whether it can be built, and one artefact that mixes all three answers none of them cleanly. Buxton (2007) adds that getting the design right is not the same as getting the right design, so a polished coded build may refine a concept the team never tested. Then bring the evidence. Walker and colleagues (2002) found low- and high-fidelity prototypes equally good at uncovering usability issues in both paper and computer media, with results independent of the medium, and Virzi and colleagues (1996) found substantially the same problems uncovered in both conditions, so the polished build has no demonstrated advantage as a test. Sauer and Sonderegger (2009) show that fidelity and visual aesthetics shape what participants report, so the team's finish risks biasing the very evidence it wants. What I would change is the choice of prototype by question: build the cheapest artefact that can answer each question, run a low-fidelity version early to diverge, and reserve implementation prototypes for questions about feasibility. Chen and Yoon (2024) record that current practice separates a prototype's fidelity from its medium and its dimension, and Codner and Lauff (2023) show that what an audience understands depends on signifiers and affordances, not on finish alone, so the team should name the audience it is designing the test for. Finally, name the experiment that would settle it: matched prototypes at one declared fidelity, the same participants, and a measure not only of the problems found but of whether the prototype changed a decision.

Begin by naming the assumption. The plan assumes that participation is a supply of feedback: applicants react to a proposed design, and a workshop is enough to make the design participatory. Then state the unit's objection. Bjerknes and Bratteteig (1995) separate participation, which is involvement, from influence, which is the power to make design decisions, and the plan transfers no power, because the supplier finalises the design afterwards. Bratteteig and Wagner (2016) show that decisions are interlinked, so a workshop held after the triage rules are drafted cannot reopen the earlier choices that decide who is ranked first. Ehn (1993) supplies the standard the plan fails, since democratic participation and skill enhancement were objectives of design rather than a courtesy offered at the end. Vines and colleagues (2013) explain the mechanism: participation is configured, and a configured workshop can legitimise a decision that has already been taken. Bring the evidence. Bossen and colleagues (2016) found that evaluations of the field's aims are rare and led by researchers rather than participants, so the council should not expect the workshop to demonstrate empowerment on its own. Steen (2013) warns that the quality depends on the practitioners' dispositions, which is why the commitment has to be written into the procurement rather than left to the facilitators. What I would change is the object of participation: give applicants authority over the triage criteria and the appeals route, not only over the interface, and open the supplier's draft rules to challenge before they are fixed. Finally, name the study that would settle it. Delgado and colleagues (2023) provide the frame, asking whether an approach grants people substantive agency, and Agnew and colleagues (2024) show what is lost when participants are treated as substitutable. The study the field needs measures two things separately: what applicants were involved in, and whether they held power over a decision. The council should run that measurement before it signs.

Begin by separating the two claims the business case runs together. The first is that co-design changes who authors a design: Sanders and Stappers (2008) define it as collective creativity applied across the whole design process, and a single workshop held after the specification is drafted is consultation, not that. The second claim, that co-design improves clinical outcomes, is the one the unit's evidence does not support. Sumner and colleagues (2024) found that none of 17 co-designed remote monitoring projects evaluated the efficacy of its own technology, reporting usability, usefulness, feasibility and satisfaction instead. Greene Barker and colleagues (2024) pooled 20 co-designed mental health interventions and found non-significant effects on depression and anxiety with low confidence by GRADE, and Slattery and colleagues (2020) found that the effectiveness of research co-design had rarely been tested empirically at all. The claim that is defensible is the one Steen and colleagues (2011) make: benefits for the project, the users and the organisation, which are real but not clinical. Brandt and colleagues (2012) then explain why one workshop cannot carry the claim, since participatory design is a family of practices rather than a single method, so the intervention would have to be specified before it could be evaluated. What I would change is the design of the procurement. Give patients authority over the monitoring rules and the escalation route rather than only over the interface. Write both a clinical outcome and a participation measure into the contract before any pilot. Pre-register that outcome with enough statistical power to detect the effect the trust claims. Farr (2018) supplies the reason for measuring power alongside the outcome, since collaborative techniques can reproduce existing power relations as well as challenge them, and a fall in falls attributed to co-design without that measure would be a claim the evidence cannot yet support.

Separate the two claims the business case runs together. The first is that values were elicited and a trade-off was made explicit. The second is that the system will support fairness, which is a claim about what the deployed artefact does. The unit supports the first as a description of method and not the second as a claim about effect. Friedman, Kahn, Borning and Huldtgren (2013) give VSD its structure of conceptual, empirical and technical investigations, and their point that technologies provide value suitabilities is precisely that architecture decides which values are easy to support, so agreeing a value in a workshop does not fix what the scoring function will do. Winkler and Spiekermann (2018) found that only 17 of 229 contributions developed technology with the complete tripartite methodology and only four of those reported iteration between investigations, and Gerdes and Frandsen (2023) record that the absence of technical analyses, reflections and solutions is a problem for the framework. So the artefact is the least-evidenced part of the supplier's story. Manders-Huits (2011) and Jacobs and Huldtgren (2018) add the deeper problem: without an ethical commitment, the process cannot distinguish a moral value from a stakeholder preference, so fairness may be a preference the supplier named rather than a value it can defend. Le Dantec and colleagues (2009) show why the elicitation should be treated as provisional, since values are located in lived experience and change as it does. Shilton (2018) shows why tenants cannot be the only stakeholders counted, because infrastructure affects people who never use it, and difficulty defining stakeholders is a documented barrier rather than a procedural detail. Umbrello and van de Poel (2023) show why the promise about behaviour is the weakest part of the case, since values predict behaviour poorly and norms predict it better. Before deployment I would require three things: an independent audit of which values the scoring function actually embodies, compared with those elicited; a published account of the trade-off decisions and who owned them, including the indirect stakeholders; and a live measure of outcomes disaggregated by the groups the system affects, with a route to suspend the system. The authority should treat the claim as unproven rather than false.

Reformulate the claim before judging it. 'Open public debate' assumes an audience and a change, so the assessment should ask who is reached and what alters in what they think or decide, rather than treating the phrase as self-evident. Dunne and Raby (2013) supply the ambition and the primary statement, but they also insist the objects are not deployed for people to use, which is the position against which any measured claim must be argued. Tonkinwise (2014) sharpens the audience question by arguing that a shopping framework and a taste regime shape the futures on offer, so the studio should show whose debate it expects to reach. Auger (2013) separates thinking about the future from critiquing present practice, and Malpass (2013) shows by interviews that the vocabulary is defined by intent and relation to the present; the proposal should therefore say which of these it is doing. Bardzell and Bardzell (2013) found the field's concepts and methods unclear and difficult to adopt, and Lindley and Coulton (2015) found design fiction inherently ambiguous, so the proposal must define its object. Pierce (2014) shows that a design proposal is not a prototype and is judged by different standards, which supports the unit's view that the debate claim needs its own evidence. Elsden and colleagues (2017) come closest to providing it, because speculative enactments put participants inside consequential circumstances, but their cases report what participants did rather than what the speculation changed. Wong and Khovanskaya (2018) warn that the practice entered HCI through corporate speculation, so a claim to public debate should not be assumed to be critical. Ringfort-Felner and colleagues (2025) offer nine qualities that a pre-registered evaluation could use, and Zhu and colleagues (2024) show that artefacts play different roles, so the panel should ask which role is claimed. Before accepting the claim I would require a pre-specified audience, a stated route to reach them, independent coding of change against a non-speculative comparison, and the criteria of Ringfort-Felner and colleagues (2025) fixed in advance.

Reformulate the claim before judging it. 'Which design approach users prefer' is a claim about a population, and the unit's thesis holds that design research does not sample one, so the student is asserting a population-level preference from three artefacts. Frayling (1993) supplies the distinction between research through, about and for design, and the project should say which it is: documenting what each prototype does is research through design, whereas a claim about user preference would need the sampling of research about design. Zimmerman, Forlizzi and Evenson (2007) offer four lenses for evaluating a contribution, and Fallman (2008) shows that design practice, design studies and design exploration are judged by different criteria, so the student should name the corner of the triangle the claim sits in. Gaver (2012) warns that the theory will be provisional, contingent and aspirational, which caps the claim; Gaver and Bowers (2012) and Hook and Lowgren (2012) show the knowledge is carried by annotated portfolios and strong concepts rather than by a preference ranking, and Lowgren (2013) confirms these are intermediate-level forms. Redstrom (2017) argues theory is made through designing, and Dourish (2006) warns that demanding transferable implications truncates the account, so the student should not force a population claim onto the work. Koskinen and colleagues (2011) show the setting determines what may be claimed, and Bardzell and colleagues (2016) require the process to be documented so a reader can follow it. Before accepting the preference claim I would require a sampled study with a stated population; within the design project I would require Prochner and Godin's (2022) quality dimensions, Robinson and colleagues' (2024) criteria for formative situations, and the boundary Almendra and da Silva (2025) draw between research through design and research by design.

Start by naming the assumption, which is that fairness consists in grading against a published list. The proposal treats critique as an assessment event rather than a judgement practice, and Bardzell (2011) argues criticism is a knowledge practice that produces defensible interpretation rather than a verdict, while Bertelsen and Pold (2004) show the aesthetic strand can be given handles in an interface criticism guide. So part of the proposal is well founded: some of critique's content is statable, and Oh and colleagues (2013) show its situations, roles and content can be separated for discussion. The evidence for the grading half is weaker. Uluoglu (2000) found that design knowledge travels through general and personal attributes, and the personal ones are usually left unstated, so a rubric that captures only the general attributes will reward the visible half of the reasoning. Dannels and Martin (2008) found that critique feedback socialised students into identities closer to academic stages or an idealised workplace than to professional practice, which warns that any rubric teaches a working culture along with a standard. Gray (2013) adds that critique reproduces the studio's hierarchies while widening participation. Smith (2024) supplies the direct observation: reviewer contributions dominated design review talk and students rarely took the initiative, even though reviewers asked questions and built trust, so the problem is the form and its authority rather than the reviewer's manner. Nyboer's (2024) students, by contrast, wanted frequent personalised guidance rather than gatekeeping. The costs follow. Schon (1983) locates professional knowing in reflection in and on action, and Nelson and Stolterman (2012) call design judgement practical wisdom, so a rubric may train compliance with a description of judgement rather than judgement itself. Sengers and Gaver (2006) argue several interpretations can coexist, so a single criterion set will grade a legitimate reading as a failure, and Lowgren and Stolterman (2004) put criteria in what a designer will defend, which keeps them arguable. Before adopting the rubric I would require three things: that it cover interpretation and grounds rather than only visible features; that it be assessed by blind agreement between two critics on the same piece, since the unit's criterion position predicts such agreement; and that the studio record whether the rubric narrows the range of readings students defend, as the meaning-cost position predicts. I would also run Smith's (2024) interaction analysis before and after, because the unit predicts criteria change who talks. Connor and Irizarry (2015) show that practitioners treat critique as a facilitated skill with preconditions, which is the part of the proposal worth keeping: publish the process, pilot the rubric, and test it rather than assume fairness is produced by publication.

A strong answer separates the two features, because they have different budgets. Suggestions need a structure keyed by prefix: a trie over product names, or a prefix index if the catalogue is largely static, returning the few best matches for the typed prefix. The cost to predict is proportional to the length of the prefix and the number of suggestions returned, not to the size of the catalogue, and the space to predict is the size of the index, which is the term to check, because an unpruned trie or a prefix map can be many times larger than the catalogue itself. The results page should not sort 400,000 rows in the application. If price is indexed, the same query can read a page of rows in order, and the prediction is that read cost follows the page size rather than the catalogue size. Measurement should record the catalogue size, report p95 and p99 latency for a keystroke and for a page load rather than the mean, and state whether the index was warm. Queries drawn from real logs are better evidence than invented ones. The limit is the important part: the measurement cannot say whether the latency is acceptable, because that depends on the purpose the response serves, and it cannot show behaviour at a catalogue size nobody loaded.

A mean over 1,000 queries is a start rather than a finding, for four reasons. The distribution matters more than its centre: a mean of 12 ms can sit alongside a ninety-ninth percentile of two seconds, and it is the slow queries that users remember. The query mix matters: 1,000 synthetic queries may all be short and common, while real use includes rare long prefixes that were never exercised. The state matters: a warm cache against an empty index is a different system from a cold one under concurrent load, and a single process run hides the contention and the collection pauses that production produces. The scale matters: 12 ms at 1,000 records says nothing about 400,000, which is the size a prediction would have been about. What would settle it is the same measurement reported as percentiles over a sample drawn from real query logs, at several data sizes, with the hardware and the warm or cold state stated, and paired with the predicted curve so that any disagreement can be attributed to a cause. Until then, the honest position is that the number establishes that the feature is fast on that machine for that query set, which is worth having and is not the claim that was made. Two things this kind of measurement can never supply: whether the latency is acceptable for its purpose, and whether the structure choice still holds at the next order of magnitude.

State the bound first in the terms its authors would use: for example, that a hash-table lookup is expected constant time and a sorted-array lookup is O(log n), so the table is preferred when keys are many and arrive without order. Then name the model, using Hartmanis and Stearns (1965), who measure the complexity of a computation by how fast a chosen machine can produce its output, so that the bound is a relation between the problem and the machine rather than a property of the problem. Use Knuth (1976) to say what the bound omits: it hides a constant, and an upper bound says nothing about typical behaviour, so an expected constant-time lookup on user-chosen keys is a claim about a distribution. Use Cormen, Leiserson, Rivest and Stein (2022) to say which case the bound names, since an expected-case bound is only as good as the model of how keys arrive and an amortised bound is a claim about a sequence of operations rather than one of them. Then take a side on the contested question. If you argue the bound predicted the observed behaviour, defend it as Cook (1983) would, treating the proved bound as the answer and the timing as an implementation detail. If you argue that it did not, use McGeoch (1996) and Johnson (2002) to say that a bound predicts nothing until it is measured on the inputs and the machines the model assumes. Close by naming the evidence that would show the bound wrong about the interface, such as a measured upper percentile of per-keystroke latency under adversarial keys that no constant could explain.

State the two bounds in the machine the bound assumes: for example, that a traversal of a contiguous array and of a linked list are both linear in the number of elements, so a uniform-cost model ranks them equal, and that a hash-table lookup is expected constant time while a tree lookup is logarithmic. Then set the terms with Denning (1970), who named locality of reference as the property that makes a hierarchy work and formalised the working set as the pages referenced in a recent window, and with Aggarwal and Vitter (1988), who replaced the operation count with transfers of blocks between an internal memory of size M and a slower store, and whose sorting bound shows that a change of cost model changes the ranking rather than merely its constant. Give the magnitude with Drepper (2007), who measures an L1 hit at about 3 cycles against a main-memory access at about 240, or with Gray and Shenoy (2000), who report a disk access of about 10 milliseconds and decompose a random access into seek, rotation and transfer, which a sequential transfer avoids. Then name the mechanism with Falsafi and Wenisch (2014), who show that prefetchers reward simple and regular access patterns and are baffled by pointer chasing, or with Chilimbi, Hill and Larus (1999), whose clustering and colouring gave 42 per cent to RADIANCE and 27 per cent to VIS without changing a single operation count. The argument to make is that the equal ranks of the linear bounds do not survive the hierarchy: the contiguous traversal is prefetchable and the linked traversal is not, so the bound has ranked two different machines as if they were one. Close by taking a side. If you argue the hierarchy is a structural constraint that a designer must build around, use Wulf and McKee (1995), who projected 1.52 cycles per memory access in 2000 rising to 98.8 in 2010 and found a five-cycle wall within eleven or twelve years even at a 99.8 per cent hit rate. If you argue the constraint must be re-examined per application, use McKee (2004), who asks for which applications the wall binds and argues for co-designing systems, memory technologies and algorithms together.

Begin by separating the interface from its representation, as Aho, Hopcroft and Ullman (1983) do: name the operations the interface performs often and say what each must cost, rather than assuming that the word editor or history fixes a structure. For a navigation history edited at both ends, Kaplan and Tarjan (1999) show that a purely functional deque with catenation performs push, pop, inject, eject and catenation in worst-case constant time, which is the operation set to check the design against. For an editor whose document is cut and spliced, Hinze and Paterson (2006) show that one finger tree serves as a sequence, a priority queue and a search tree, with amortised constant-time access to the ends and logarithmic splitting. Then argue the guarantee. Tarjan (1985) supplies the amortised measure and defends it as realistic for a sequence of operations, while Hood and Melville (1981) show the same structure can be made real-time, so if the interface redraws within a frame the per-operation bound is the one that matters. If versions persist, Okasaki (1998) shows the amortised bound is not stable and a redesign is needed. If several users edit together, name the correctness condition first: Herlihy and Wing (1990) define linearizability, Michael and Scott (1996) give a non-blocking queue measured under contention, and Prakash and Knister (1994) show that multi-actor undo handles conflicts rather than popping a stack. Finish by naming the workload you would measure and the per-operation latency tail you would report.

Begin by classifying the key source rather than the data structure, which is the whole point of the unit: name the keys the table stores and say who can choose them, since a key set generated internally and a key set typed by a user require different guarantees even when the interface looks identical. For a cache keyed by request parameters, the keys are attacker-influenced, so the constant-time claim does not hold as stated. Carter and Wegman (1979) show why: the expected bound holds for any input only because the function is chosen at random and independently of the keys, and a table whose function is fixed and visible has no such promise. Crosby and Wallach (2003) measure what happens when the assumption fails, and their repair is the one to adopt, a procedure for choosing the function that keeps a keyed table inside the expected bound, with Aumasson and Bernstein (2012) supplying a concrete function designed for hash-table lookups protected against hash flooding. If instead the interface must answer without any distributional assumption at all, the alternative is a worst-case structure: Fredman, Komlos and Szemeredi (1984) for a static set in linear space, Dietzfelbinger and colleagues (1994) for a dynamic dictionary with constant worst-case lookup and a stated lower bound, or Pagh and Rodler (2004) for the same worst-case lookup without perfect hashing. If the question is membership rather than exact lookup, Bloom (1970) governs and the unit's error is one-sided, with Fan and colleagues (2014) supplying deletion and Bender and colleagues (2018) showing that a per-query guarantee against an adversary can be had at no asymptotic cost. Close by naming the measurement: classify the key source, then report the distribution of per-operation latency, and of false positives for a filter, over a query sequence under an adversary who knows the implementation but not the key, because no source in this literature reports that tail.

Begin by naming the workload rather than the structure, which is the whole point of the unit: give the ratio of reads to writes and say who writes concurrently, because two interfaces that both expose an ordered map require different guarantees once those numbers differ. For an append-only event store that is read rarely and written constantly, the read-to-write ratio is very low, and the unit's evidence points away from a comparison-balanced tree: O'Neil and colleagues (1996) show that maintaining a B-tree index in real time effectively doubles the transaction's I/O cost and raises total system cost by up to fifty per cent, and their log-structured merge-tree instead defers and batches index changes across sorted runs, which is the same choice a modern store makes with the read amplification Luo and Carey (2019) catalogue. If the interface is versioned, as an editor with undo history is, the write pays again, and Driscoll and colleagues (1989) give the persistent forms that keep logarithmic access and constant space per update, so the argument must price the old version the write must preserve. If the tree must instead answer for any key set with the bound independent of the data, the invariant is the right answer: Adelson-Velskii and Landis (1962) buy the logarithmic read with rotations on update, Nievergelt and Reingold (1973) put the read against the write on an explicit parameter, and Guibas and Sedgewick (1978) show the schemes share one analysis, so the choice is the dial rather than the bound. Where the bound must hold for a memory the structure cannot measure, Bender and colleagues (2000) give the cache-oblivious layout, and where the invariant is replaced, Pugh (1990) or Sleator and Tarjan (1985) give probabilistic and self-adjusting alternatives. If many threads write, the concurrency model enters: Blelloch and colleagues (2016) reduce the ordered set to one parallel join and find four balancing schemes performing similarly, so the invariant stops deciding. Close by naming the measurement: report the read-to-write ratio actually observed, the tail latency of both reads and writes rather than the mean, and, if a learned model is proposed, the same tail under a shifting key distribution, since that is what would separate Kraska and colleagues (2018) from the invariant.

Begin by naming the query rather than the structure, which is the unit's whole point. A map viewport at low dimension is a highly selective orthogonal range query, so the pruning argument holds and an R-tree (Guttman, 1984) is the right structure, with the R*-tree's combined optimisation of area, margin and overlap (Beckmann and colleagues, 1990) improving the margin that the query depends on. Say where the index's cost still lands: every insertion maintains the bounding rectangles, so a read-heavy viewport pays little and a write-heavy edit stream pays more. For image similarity or recommendations, the objects are feature vectors of high dimension and the query is a nearest-neighbour search rather than a range, so the unit's evidence points the other way. Bentley (1975) shows the tree bounds degrade with dimension, Berchtold and colleagues (1996) show directory overlap reaching about 90 per cent by dimension five, and Weber and colleagues (1998) show that under stated assumptions a sequential scan wins beyond around ten dimensions. If exactness is non-negotiable, a scan may genuinely be the honest answer. If a near neighbour will do, the query is redefined and the index pays again: an approximate index such as hierarchical navigable small world graphs (Malkov and Yashunin, 2020), or the locality-sensitive hashing of Indyk and Motwani (1998) that Andoni and Indyk (2006) brought near the lower bound, recovers sublinear time at a stated recall. If the data admit only a distance function, an M-tree (Ciaccia and colleagues, 1997) or a cover tree (Beygelzimer and colleagues, 2006) applies, but the cover tree holds only under a bounded expansion constant, which is an assumption about the data that must be stated. Close by naming the measurement: the number of distance computations and page input and output operations, or the latency, as a function of intrinsic dimension and query selectivity, on data whose distribution is stated, comparing the chosen index, a scan and, where the answer may be approximate, the approximate index at matched recall. A design claim that does not report those numbers is a claim about the query model rather than about the workload.

Begin by naming the promise rather than the structure, which is the unit's whole point. A notification list ordered by score is a repeated extraction of the maximum, so the relevant contract is Williams's (1964) binary heap: the extreme element in constant time and a logarithmic insertion or removal, without a full order. Say whether the interface can display equal keys, because that is what decides whether the indifference matters. A leaderboard with tied scores can, so the order the user sees is determined by the heap's internal layout and by the sequence of insertions and removals that produced it, and two tied entries can change places between refreshes even though nothing about them changed. If the interface needs only the next item and never displays the ties, as a scheduler that runs the highest-priority job does, then the indifference is invisible and selection rather than ordering is the appropriate frame, which is Hoare's (1961) argument that one element is often all that is wanted. If the ties are displayed, supply the total comparison that the structure does not: a secondary key such as an arrival counter makes no two distinct items compare equal, and the extracted sequence becomes deterministic. Note that no faster heap removes the need, since Fredman and Tarjan (1987) and Fredman and colleagues (1986) define operations that say nothing about equal keys, and Johnson (1975) shows that the operation mix, including whether decrease-key is used, is what decides the structure. Close by naming the measurement: wallclock latency plus element comparisons and cache misses on the interface's own operation mix and key distribution, following Larkin and colleagues (2014), and a check of the displayed sequence's stability across repeated runs, which is the quantity no source here measures.

Begin by fixing the question, because the unit's claim is that the question does not choose the algorithm; the model does. A route planner that returns the quickest route between two places asks a single-pair shortest-path question, and its edge model is a graph whose costs are travel times that never decrease, so the method Dijkstra (1959) gives applies, with its settled set safe because no cost is negative. Say what would break it: introduce a cost that may be negative, as a discount or a credit on an edge, and the assumption is gone, so the honest choice is Bellman's (1958) relaxation, which finds the route and reports a negative cycle where the cheaper loop never closes. If the interface instead needs the distance between every pair, the model changes the object of the answer, and the relaxation of Floyd (1962) is the algorithm for that object rather than a faster way to answer the single-pair question. Where the graph is large and static, Geisberger and colleagues (2008) show that the fastest query may run on a shortcut hierarchy the algorithm built rather than on the input, so the speed is a property of the constructed model. Where the planner is guided by an estimate of the remaining distance, Hart, Nilsson and Raphael (1968) make the guarantee a property of that estimate: the search is optimal only while the estimate never exceeds the true remaining cost. Close by naming the measurement rather than an expectation, because Cherkassky, Goldberg and Radzik (1996) found that the strengths and weaknesses of the algorithms shift across problem classes, so the answer returned, the resources spent and the instance family must be reported together. The model, named and checked, is what licenses the claim.

Begin by naming the distribution, because the unit's claim is that the worst-case bound and the typical case diverge. An activity log ordered by time meets nearly sorted data, so the honest description is adaptive rather than worst case: Bender, Farach-Colton and Mosteiro (2006) show that insertion sort with gaps left between items runs in n log n with high probability, and Estivill-Castro and Wood (1992) make the disorder of the instance part of the cost. A list sorted on a column with many equal keys meets the case Bentley and McIlroy (1993) handle by partitioning equal keys in the style of the Dutch National Flag, and a sort that the user re-runs on every keystroke meets an interface's time budget, where the constant and the branch behaviour decide the outcome that Edelkamp and Weiss (2019) measure. Where the data is numeric with a known shape, say so, and say that the learned sort of Kristo and colleagues (2020) fits a model of the distribution and refines locally, which is why the comparison bound does not describe it. State the bound the textbook would quote, then say what the shipped sort adds: Musser (1997) limits the partition depth and falls back to heapsort so the quadratic worst case cannot occur, and Peters (2021) detects and defeats adversarial patterns while running in linear time on a few common ones. If the user can choose the keys, treat the input as adversarial and rely on that detection rather than on the expected case. Close by designing the measurement rather than asserting an expectation: fix a set of input distributions, measure the comparisons, the moves, the branch or cache behaviour and the wall-clock time for a small and a large input, and report the worst run as well as the typical one. Katajainen and Pasanen (1999) show why the choice of cost matters, because an algorithm judged on moves rather than comparisons can win or lose depending on which is counted, and Wild and Nebel (2012) show that a library decision made on measurement deserves an analysis to explain it.

Begin by naming the task, because the unit's claim is that the budget belongs to it. A mail search over a large mailbox is used in the middle of other work, so a suggestion that arrives two seconds late has broken the person's thread, while a background export can take much longer without harm; Miller (1968) is the source for treating acceptable delay as a property of the activity rather than a constant, and Card, Moran and Newell (1983) show why a quoted constant hides a range, giving their cycle times as estimates. Next separate the two operations. The exact part should be answered from an index, following Ferragina and Manzini (2000) for a structure whose space follows the entropy of the text, or Knuth, Morris and Pratt (1977) for a single linear scan of pattern and text. The fuzzy part should never compute a distance against every message: Backurs and Indyk (2015) show that edit distance is unlikely to be computable in strongly subquadratic time, so the pipeline Broder (1997) describes is the right shape, narrowing the corpus with a cheap sketch and re-ranking only the survivors, whose closeness Levenshtein (1966) defines. Finally design the measurement rather than asserting an expectation: fix a mailbox size, record the latency at the 50th and 95th percentile for a warm and a cold run, and check the ranking against a set of misspelled queries that a person would expect to match. Treat acceptability as part of the result, since MacKenzie and Ware (1993) and Ng and colleagues (2012) both show that a change in latency alters what a person can do, not only how long they wait.

A strong answer separates three decisions rather than describing one screen. The relational core holds modules, departments, terms and the offerings of a module in a term as separate records, because each is a thing that has to be counted and joined in its own right. A student's enrolment is its own record with keys to the student and to the offering, which is the same argument as the order-line case in the narrative. The facets are exactly the fields a filter offers, so department, level, term and assessment type need controlled vocabularies rather than free text. A vocabulary with thirty values is usable and one with three hundred is not, so the level field is the place to test whether a vocabulary is doing any work. Keyword search and faceting do different jobs: the inverted index handles the long tail of a module title, and the facets handle the question a student usually asks, which is a combination rather than a phrase. Evaluation cannot rest on intuition. Assemble real queries from search logs, have two people judge the top ten results for each, and report precision at the top ten together with the agreement between judges. State plainly that recall is not measurable unless the relevant set is enumerated, which for a catalogue of this size it usually is not. Then name what would change the design: if sessions are dominated by facets and almost never use keywords, browsing should be the default route and the search box the fallback.

The temptation is to run one method and call the output a finding. A defensible programme states what each method can support before it is run. Card sorting generates a candidate structure from the way people group products, and it can be run quickly with a modest number of participants. The published case study on a retail site found that eight to twelve participants gave reliable closed card sort data, so a week is enough for a first pass. Card sorting cannot tell you whether a label works in the interface someone actually sees, because participants are reacting to cards rather than to a page. Tree testing fills that gap. Participants hunt for items in a text-only hierarchy, so labels and grouping are tested without visual design, and recent work comparing tree testing variants against high-fidelity prototypes found that the variant showing earlier navigation choices predicted real navigation most accurately. Neither method measures demand, and that is the third piece. Search logs show what people typed and whether they found anything, which is evidence about the volume of a need rather than the quality of a label. The report should therefore state one decision, the evidence for it, the gaps, and the cheapest next measurement. The contested question deserves a line as well: the taxonomy will be tempted to follow the retail organisation's buying departments, the evidence leans towards the shopper's task, and the reader should be told which one was chosen and why.

Begin with the model's stated purpose. Codd (1970) aims to protect users of large data banks from having to know how the data is organised inside the machine, so that terminal activity and most application programs survive a change in the internal representation. That gives the essay its test: a fact belongs inside the schema when the model can name it as a value and constrain it there. Codd (1979) supplies the limit from the model's own author, who extended the relational model to capture more meaning because the bare model did not name every fact an application needed. Use him to make the claim concrete. If a rule about the fact cannot be written as a dependency or a key, then the constraint lives in application code, and a change in that code is not covered by the independence claim. Aho, Beeri and Ullman (1979) sharpen the schema side, because naming a fact as a relation also creates the obligation to show that the relations recombine losslessly. A schema that hides this obligation can return more rows than the facts justify, so moving the fact inside the schema is not costless. Then take a side. Kim (1990) argues that the relational type system was too weak for complex objects, which supports reading the boundary as a real limit; Stonebraker and Pavlo (2024) argue that SQL absorbed the alternatives and that the relational systems remain where the advances happened, which supports reading it as a movable boundary. The closing paragraph states the evidence that would distinguish the two positions, such as the cost of a schema change weighed against the cost of the application code that enforces the same rule.

Begin with the fact. Name the system, the fact and its copies, and say which copy is authoritative. Chen (1976) supplies the first claim, that naming entities and relationships is a design act recorded in a diagram, so the opening question is whether the fact was named at all. Fagin (1981) sharpens it, because a normal form is worth having only if it excludes a named anomaly, so the answer has to say which insertion or deletion anomaly the duplication permits. Then state the unit's two conditions. The first, from Gupta and Mumick (1999), is that a duplicated precomputed relation carries an ongoing reconciliation obligation, so the copy is sound only when one write path owns it and its maintenance is a designed and bounded process. The second is that the read path justifying the copy has been measured, and Kimball and Ross (2013) carry the case for denormalising from the side that made star schemas standard. Next, take a side. Argue that the duplication is sound or that it is debt, and connect the judgement to whichever condition fails, rather than listing both conditions and stopping. Close with the evidence, such as a measured query cost against the total reconciliation cost of the copies, counting the cost of a schema change and the code held outside the database, because Curino, Moon and Zaniolo (2008) show that schema change remains error-prone and frequent.

Begin by naming the system and the query, and say who waits for it, so the cost has a stake. Then state the contract from Chamberlin and Boyce (1974), that the language fixes what can be asked and leaves how it is answered to the system, and from Selinger and colleagues (1979), that the user supplies predicates and the engine selects the access path. Next, state the failure. Leis and colleagues (2015) find that industrial-strength cardinality estimators routinely produce large errors, so the decisive input to the plan is the least reliable one, and the engine does not advertise the size of its error. Now turn to physical design. Name the structures you would build, such as a B+-tree index from Comer (1979) for an equality lookup or a range scan, or a log-structured merge-tree from O'Neil and colleagues (1996) if the workload is write-heavy and you can accept the read cost, and connect each structure to the access pattern it serves. If you prefer, use Abadi, Madden and Hachem (2008) to argue that the storage model alone does not explain the performance, so the mechanism must be named. Take a side: say whether you would let the optimiser choose from whatever structures exist, or fix the access paths from a workload you have read. Defend the choice rather than listing both options. Close with the measurement that would settle it: run the same queries over the same data twice, once with the optimiser free and once with fixed paths, and compare, because a plan you have not read and timed is a guess.

Begin with the system and the operation, and name the outcome you would not accept, so the guarantee has a stake. State from Gray (1981) that a transaction is a transformation of state with atomicity, durability and consistency, and from Mohan and colleagues (1992) that a recovery method such as ARIES, with write-ahead logging and partial rollback, is what makes undo and redo real. Bring in Eswaran, Gray, Lorie and Traiger (1976) for the condition your locking must obey, that a transaction may not request a new lock after releasing one. Then take a position on the level. Name the level you would set and, using Berenson and colleagues (1995), the anomaly it may still permit: a level can clear the checklist of dirty read, non-repeatable read and phantom, as snapshot isolation does, and still allow an outcome no serial order produces. Use Fekete and colleagues (2005) to say how you would decide whether your application is safe at that level rather than assuming it. Then price the alternative. Cahill, Rohm and Fekete (2009) move the cost of correctness into detection and abort, and Ports and Grittner (2012) report that PostgreSQL's serialisable level achieves performance only slightly below snapshot isolation and beats two-phase locking on read-intensive work. Say who pays: the aborted transaction, the retry, and the engineer maintaining the lock manager. If the system must stay available, use Bailis and colleagues (2013) to name the guarantee you give up, because some guarantees cannot be had while remaining available during a partition. Close with a measurement rather than a preference: run the application at your chosen level and at the next stronger one on the same workload, count the anomalies, the aborts and the retries, and compare.

Begin by naming the feature and the operation, then state the staleness it can tolerate: a product price may accept a few seconds of difference, while a seat count must not admit two buyers of the last seat. Use Brewer (2012) to say why the choice of two letters is too coarse, since handling partitions explicitly lets a designer aim at consistency and availability together, and Abadi (2012) to add that latency forces the same decision even with no partition, because a distant replica is slow to read. If the feature is made available with application-assisted reconciliation, describe the cost from DeCandia and colleagues (2007), whose Dynamo resolves conflicting versions through object versioning and application-assisted reconciliation, and from Terry and colleagues (1995), whose Bayou places merge procedures and dependency checks in the application. Then say whether a weaker guarantee would do less work: Lloyd and colleagues (2011) offer causal+, the strongest consistency achievable under geo-replication, and Shapiro and colleagues (2011) offer conflict-free replicated data types that merge without application logic. If the feature instead needs strong consistency, use Corbett and colleagues (2012) to price it, because Spanner keeps externally consistent reads and writes at global scale using TrueTime, at a cost in latency and in atomic clocks and GPS receivers. Close with a measurement: run the feature across a region failure and record the staleness window, the latency and the availability, rather than arguing from the label.

Begin by naming one feature and the operation it is read and written by, and say how fixed that pattern is: a session read and written whole suggests key access, while a permission query that walks a chain of relationships suggests traversal. Use Cattell (2011) to say what a store matched to that pattern buys, since the stores he surveys were built to scale simple transaction processing over many servers and contrast on data model, consistency, storage, durability, availability and query support, typically sacrificing some of these dimensions. Use Davoudian, Chen and Liu (2018) to show that the choice is not one decision but four coupled ones, because their survey organises the stores by data model, consistency model, data partitioning and the CAP theorem, and the principles are non-orthogonal. State what the feature gives up in exchange, which the unit names as the joins, constraints and consistency the relational model would have enforced, and the weaker concurrency Pokorny (2013) records as part of the model. Then bring evidence that the match matters: Vicknair and colleagues (2010) found the graph store faster for the query shape it was built for while the relational store stayed efficient where joins of large tables were not needed, and Parker, Poe and Vrbsky (2013) reported where each was faster on a modest structured dataset. Take a side. Stonebraker (2010) holds that the performance arguments for a non-relational store are insufficient and the relational store scales, so the bet may buy less than it costs; Meijer and Bierman (2011) hold that SQL and NoSQL are two sides of one coin, which would make the bet a difference of syntax. Close with a measurement that would decide the argument, such as a head-to-head run on the named access pattern that records not only throughput but the constraints and consistency outcomes the losing store cannot express and the cost of reproducing them in application code.

Begin with the feature and the collection it searches, and say how the results are ordered, because the unit's claim is that the index fixes what can be computed cheaply. Use Zobel and Moffat (2006) to say that because the index records, for each term, the documents that contain it, matching is a dictionary lookup and a merge of postings lists rather than a read of every document, and use Manning, Raghavan and Schutze (2008) to define recall as the fraction of relevant items retrieved and precision as the fraction of retrieved items that are relevant, with their F measure combining the two. Then state the judgement problem the same source describes: pooling asks assessors to judge only the documents the systems under test already returned, so a method that retrieves something unpooled cannot be credited for it, and relevance is a judgement a person makes at a moment rather than a property of the document. State the disagreement. Kamphuis and colleagues (2020) compared eight BM25 variants, including the Lucene implementation practitioners usually mean, on three newswire collections and found no significant effectiveness differences, which suggests the weighting details barely matter. Thakur and colleagues (2021) evaluated ten lexical, sparse, dense, late-interaction and re-ranking systems on eighteen datasets and found systematic differences, with BM25 a robust baseline, re-ranking and late-interaction models leading at high cost, and dense and sparse models often underperforming, which suggests the family matters greatly. Take a side. Robertson and Zaragoza (2009) present the probabilistic relevance framework that led to BM25, so a weight is the term's estimated contribution to the odds of relevance, and the two findings reconcile if the variants share one assumption while the families differ in theirs; Ponte and Croft (1998) and Zhai and Lafferty (2004) make a parallel case, ranking by the likelihood that a query would be generated by a document's language model and showing that how that model is smoothed decides the ranking. Close with the evidence that would count against your choice: an evaluation that varies family and collection independently, reports effectiveness alongside the judgements and pooling depth behind each collection, and shows whether an advantage survives on collections never used to fit the model.

Begin with the facility and the collection it searches, and say how the results are ordered, because the unit's claim is that the score is a property of the collection and its judgements as much as of the system. Use Cleverdon (1967) to state the design: a fixed document collection, a fixed set of queries and relevance judged once, so that a difference in score can be read as a difference in the systems. Then show why that design cannot simply be borrowed for a deployed facility. Zobel (1998) found that the relative performance of systems in the TREC experiments appeared reliable while recall was overestimated, because many relevant documents had never been found and so could not be credited, which means the facility's catalogue can report a high score while relevant items outside the pool go unmeasured. Buckley and Voorhees (2004) show the standard measures are not robust to gross violations of the completeness assumption and offer a more robust replacement, so the honest report states how the pool was built and that recall is not measurable where the relevant set was never enumerated. State the defence. Voorhees (1998) found the relative ranking of systems comparatively stable when judgements were redone by different assessors, and Sanderson and Zobel (2005) found the t-test highly reliable and far more reliable than reporting a percentage difference with no test, so a comparative claim about two orderings can survive judge variation even where an absolute score cannot. Take a side: this unit's position is that the two findings concern different quantities, and the question is which quantity the facility's ranking actually estimates. Close by naming evidence that could count against your judgement, such as an independent set of queries judged by assessors who did not see the current ranking, and say how you would report the comparison. Use Webber, Moffat and Zobel (2010) if the two ranked lists are incomplete and non-conjoint, since rank-biased overlap compares them directly and weights high ranks, or Voorhees, Samarov and Soboroff (2017) if you can partition the collection into replicates, since bootstrap ANOVA then narrows the confidence interval around the system effect and increases the number of significant pairwise differences found.

Begin by naming one system concretely and describing its metadata scheme, listing the fields and stating which of them are controlled, because the unit's claim is that those choices are claims about which distinctions matter rather than neutral bookkeeping. Use Svenonius and Arms (2000) to state why the description matters: a retrieval mechanism is only as effective as the intelligence put into the description it searches, so the ceiling on what the system can return is set before any query runs, and a field left uncontrolled moves the cost from design to every later query. Then apply the interoperability requirement. NISO (2010) fixes what a controlled vocabulary is and how its terms relate, and ISO (2011) requires a thesaurus to work with other vocabularies, so if the scheme uses local values that no other system can interpret, it has purchased cheap population at the price of reuse. Bring in the emergent side fairly. Golder and Huberman (2006) found stable regularities in the relative proportions of tags for a resource, and Halpin, Robu and Shepherd (2007) found a power-law structure in tag distributions; neither shows that the resulting terms declare relations another system can consume, which is why the unit treats this as evidence of order rather than of interoperability. Then run the reuse test from Wilkinson and colleagues (2016): ask whether a machine that did not build the scheme could find and use the descriptions without a human in the loop, and say what would have to change for it to do so. Close by naming evidence that could count against your judgement, such as a second system that consumed the scheme without a translator, or a set of users who found items with their own terms, and say how you would gather it, for example by exporting the records to a system that did not create them and measuring how much human repair was needed.

Open by naming the site and describing its top-level labels and grouping concretely, because the unit's claim is that these are design decisions that can be wrong about users, not neutral descriptions. Use Rosenfeld, Morville and Arango (2015) to state what an architecture comprises: organisation, labelling and navigation, with search alongside them. Then make the central argument. If the labels follow the institution's departments, the architecture is the organisation chart, which is the rival hypothesis that wins by default because it is already built and cheaper to defend. Test that reading against the mechanism. Pirolli and Card (1999) argue that people follow information scent, so the label at each decision point is what steers the path, and Furnas (1997) argues that a route is only navigable if it is discoverable, which a label can destroy even when the destination exists. Then state what the two methods would measure. Dougalis and Katsanos (2025) show a closed card sort recovers a shared grouping at a stated cost, reporting that 8 to 12 participants were sufficient for reliable closed card sort data, though from one e-commerce case, so a card sort would reveal whether users group the items as the site does. Kuric, Demcak and Krajcovic (2025) show a tree test predicts real navigation but that the variant matters, so the tree test would reveal whether the labels can be found. Take a side: this unit's position is that the architecture is a hypothesis and the test is a measurement whose instrument must be reported. Close by naming evidence that would change your mind, such as users who find a target readily under the organisation's labels, and name the method you would run first, a tree test on the most contested label, since it tests discoverability without a visual design to hide behind.

The feedback should start by separating the label from the work. What the study can claim as it stands is a thematic description of the twelve interviews, organised by five categories with illustrative extracts, and that is a legitimate result if the analytic procedure is reported. It cannot yet claim a grounded theory. On Glaser and Strauss's account, theory is discovered from data that has been systematically obtained and analysed through the deliberate choice and study of several comparison groups, and it involves constant comparison, the search for negative cases and an account of how categories developed. The chapter shows none of those moves, so the label adds a promise the report does not keep. To justify the label, the student would need at least a documented sequence in which early codes were tested against later data, an account of a category that changed or was abandoned, the specific case that did not fit and what it forced, and a statement of what the resulting theory explains and predicts. The choice between the two approaches should follow the question rather than the fashion. If the aim is to describe patterns of meaning across the interviews and to interpret them in relation to the student's position, reflexive thematic analysis fits, provided it is reported as such, including which version was used and why. If the aim is to generate a conceptual account of a process with explanatory ambition, grounded theory is the suitable frame, and it will take longer. Either way, the report needs one honest paragraph on what could have been observed that would have contradicted the findings.

The critique should begin by asking what the figure is a figure about. If the authors coded manifest content with a well specified codebook, high agreement shows that the frame was communicable and applied consistently, which is a real and modest achievement. If the codes require interpretation of latent meaning, the figure mostly shows that coders were trained into a shared reading, because agreement is produced by shared training as easily as by a well defined concept. O'Connor and Joffe make the point that the disagreements are often more informative than the score, and they warn that a number can create false precision, since research shows that even nonsensical mathematical information can inflate judgements of a report's quality. Braun and Clarke (2021) go further and argue that coding reliability measures are promoted as universal quality requirements when they belong to one iteration of thematic analysis among several. The paper's sentence therefore overreaches: it presents one tradition's criterion as the criterion. A reader would need the codebook itself, with definitions and boundaries, an account of how the frame changed during analysis and what prompted each change, the proportion of data double coded and how it was selected, the position and training of the coders, the treatment of contradictory cases, and a reflexive statement of what the analyst's commitments did to the reading. Rigour would then be argued in the terms of the method used, not asserted by a coefficient.

Begin with the object of study. Ochs (1979) argues that transcription is a theory of the data, so the first move is to state what the question needs the record to carry. Name the interaction and the question it will answer, then choose the feature that question requires. Jefferson (2004) supplies the apparatus: overlap brackets, latching and gap notation, timed silence measured in tenths of a second, and marks for intonation, stress and volume, each introduced with the phenomenon it was invented to display. A study of turn-taking needs the timing notation, while a study of a shared screen needs the gesture and the gaze that a speech-only record drops, as Bezemer and Mavers (2011) argue. Then take a side. Poland (1995) treats the quality of the record as a measurable property that checking can improve, which supports arguing that the transcript is good enough once its conventions are stated. Lapadat and Lindsay (1999) treat transcription as theory laden, which supports arguing that no convention removes the analyst's commitments. Close by naming the evidence that would distinguish the two positions: a comparison of the claims two analysts draw from differently transcribed versions of the same recording, which Davidson (2009) shows the literature does not yet contain. A strong answer states plainly which features the chosen transcription preserves and which it drops, because that statement is what makes its selectivity available to a reader.

Begin with the codebook entry, because the unit's thesis is that the entry is where the decision becomes inspectable. Name the code, define it in one sentence, and then state the boundary: which segments look like instances but are excluded, and why. Ryan and Bernard (2003) show that theme identification is fundamental yet rarely described, so a boundary that is left implicit cannot be applied twice. Give two example segments, one clear instance and one borderline case, since the borderline case is what tells a reader where the edge of the code runs. Then take the provenance question. Fereday and Muir-Cochrane (2006) argue that inductive, data-driven coding and deductive, theory-driven coding can be integrated, so the honest answer is often that the code name came from the literature while its boundary was settled by the data. Use Saldana (2021) to argue that the code is analytic and that the analytic memo records why the pattern is claimed, or Bradley and colleagues (2007) to argue that the type of code chosen decides what the analysis can produce. Then take a side on the contested question. Against St. Pierre and Jackson (2014), who hold that coding forecloses theoretical work, set MacQueen and colleagues (1998), for whom the codebook is what lets another person examine the interpretive basis of the analysis. Close by naming the evidence that would separate the two explanations of your own coding: a second coder working only from your codebook, whose disagreements would show whether the boundary or the expectation drove the code.

Begin with the instrument, because that is what the figure describes. State what counts as a unit of coding and whether coders selected their own segments or were given predetermined ones, which is the decision Kurasaki (2000) shows can change the figure before any disagreement about meaning arises. Name the coefficient and the software, since Popping (2010) and Hayes and Krippendorff (2007) disagree about which measure should be standard, and a reader cannot compare .55 with published values without knowing that the team chose kappa and why. Report an interval as well as the point estimate, following Zapf and colleagues (2016), who found the asymptotic interval for Fleiss' kappa unreliable and recommended bootstrap intervals. Then report the process: state the codes that drew low agreement, because Burla and colleagues (2008) found that low rates flagged codes defined too broadly and that clarifying them improved the scheme, and Hruschka and colleagues (2004) show that codebook revision and recoding can lift most codes towards a kappa near 0.8. Say plainly what .55 does not show: it is not evidence that the themes are true, only that the team's application of the categories is not yet settled. Take a side on the contested question. Morse (1997) would read the low figure as a warning that the analysis should not be flattened to raise it, while Hayes and Krippendorff (2007) and O'Connor and Joffe (2020) would treat the figure as a precondition of shared constructs; the defensible position for a report is theirs, with the qualification that the number documents the codebook rather than the interpretation. Close with the evidence that would separate the two explanations: a revision of the two disputed codes, recoding by both researchers, and a second agreement figure, which together would show whether the difficulty lay in the definitions or in the accounts the analysts can share.

Name the assumptions before objecting to them. The manuscript assumes that thematic analysis is one method whose name is enough to describe it, that an agreement figure demonstrates the quality of an interpretation, and that themes can be reported as things that emerged from the data. Each assumption is answerable from the unit's sources. Braun and Clarke (2021) state that thematic analysis refers to a cluster of sometimes conflicting approaches, and make assuming that it is one approach the first of ten common problems they find in published work claiming to follow their guidance. The manuscript cites Braun and Clarke but does not say which version it is doing, so a reader cannot tell whether the criteria in play are those of reflexive thematic analysis or of a coding reliability approach. The kappa is the clearest sign of the second, since Braun and Clarke (2021) describe coding reliability measures promoted as universal requirements of quality as an institutional form of the homogeneity assumption, and a reflexive analysis asked for agreement between coders has been asked for the wrong thing. Kiger and Varpio (2020) show why the author might have thought otherwise, since their guide presents thematic analysis as one widely used method with a commonly used six-step framework that suits a variety of epistemological frameworks; that is the position the manuscript has absorbed without stating it. Byrne (2022) supplies the alternative in practice, showing themes produced by organising codes around a central organising concept the researcher interprets from the data rather than discovered by coders. Finlay (2021) and Nowell and colleagues (2017) set the standard the review should hold the author to, since Finlay proposes rigour, resonance, reflexivity and relevance as criteria and Nowell and colleagues ask for an auditable decision trail that lets a reader see how the analysis was conducted. Vaismoradi and colleagues (2013) add the boundary question, because a paper reporting counts should consider whether it is doing content analysis, whose criterion of difference they locate in the opportunity for quantification. What I would ask the author to change is therefore three things: name the version of thematic analysis and its assumptions, either drop the kappa or else state what it was used for and report what the coders disagreed about, and rewrite the finding so that the themes are presented as products of the analyst's organising work rather than as objects that emerged. Braun and Clarke (2023), writing as editors, show that this is the standard applied to submissions, and it is the same standard a student conference should apply.

The chapter assumes four things: that grounded theory is one method, that its procedures can be combined freely, that emergent themes are an acceptable finding, and that saturation is settled by a count of interviews. Each assumption is answerable from the unit's sources. Kenny and Fourie (2015) argue that the classic, Straussian and constructivist traditions have diverged to the point of being neither homogeneous nor interchangeable, and locate the incongruity in contending coding procedures, opposing philosophical positions and conflicting uses of literature. Vollstedt and Rezat (2019) show what the coding paradigm is, the instrument that organises a Straussian analysis, so pairing it with Charmaz's constructivist commitments is a contradiction rather than a synthesis. Charmaz (2006) treats a category as something the analyst and the participants produced together, so it cannot be reported as having emerged. Heath and Cowley (2004) give the underlying difference: whether theory emerges from the data or is shaped by the researcher's interpretation. Strauss and Corbin (1990) supply the procedural template the chapter has partly followed, and Corbin and Strauss (2015) show that the template itself has moved towards the analyst's interpretive role, which is why the mixture is understandable but still has to be declared. Howard-Payne (2016) shows that choosing a version is a decision made before the analysis and defended within it. Timonen and colleagues (2018) offer the possible defence, a shared core of procedures usable from any epistemological position, and the chapter could adopt it if it said so. Charmaz and Thornberg (2021) show the cost of not choosing, since quality criteria are given form by form. My requirements: name one programme and state its assumptions, align every procedure with it, replace the discovery language with an account of the analytic work, and justify saturation through theoretical sampling of the kind Glaser and Strauss (1967) and Vollstedt and Rezat (2019) describe rather than by interview count.

The analysis assumes four things: that a keyword count is a sufficient measure of frustration, that reliability settles what the count shows, that the interpretive step may be left implicit, and that the clustering in two weeks is evidence about users rather than about the vocabulary of a support queue. Each assumption is answerable from the unit's sources. Hsieh and Shannon (2005) place this design in the summative approach, in which counting keywords or content is followed by an interpretation of the underlying context; the interpretation is part of the method, so the conclusion that frustration rose is not a finding until it is argued. Neuendorf (2017) shows that the reliability figure is a condition of measurement, since coders must agree where a unit begins, and Kleinheksel and colleagues (2020) set out further threats to reliability and validity, so the coefficient answers one question and does not certify the conclusion. Weber (1990) makes the deeper point that classification is the analytic step: a keyword list decides which differences can be seen, and sorry cannot distinguish a formula used every few lines from an apology that repaired a failed turn. Graneheim and colleagues (2017) supply the criterion the team has to meet, the demonstration of the logic by which categories were abstracted and connected to the aim, and they warn that increased abstraction and interpretation raise the burden of demonstrating credibility. Lindgren and colleagues (2020) add that abstraction and interpretation must be examined where meaning units are selected and coded and where categories are built, not at the end. The repair is therefore to treat the count as an input: define units and categories with reasons, code recognisable meaning units rather than three words, and write the interpretation as a claim that can be contested separately from the distribution, using the phases Elo and Kyngas (2008) set out and the planning-stage attention to credibility that Bengtsson (2016) recommends. Downe-Wamboldt (1992) shows what the team was reaching for, since content analysis aims at external validity, and Mayring (2000) offers the rule-guided procedure that would make the move repeatable. What would strengthen the work most is a second reading of the messages the keyword count could not separate, together with the variant declaration Vears and Gillam (2022) ask of beginners, so that a reader can see which content analysis was done and why.

The analysis assumes four things: that a turn carries its meaning as a content property, that the same word does the same action wherever it appears, that a reliability coefficient establishes that the codes measure confusion, and that the timing of the assistant's clarifying question can be read from the user's coded turn. Each assumption is answerable from the unit's sources. Sacks, Schegloff and Jefferson (1974) show that turn taking is locally managed, party-administered, interactionally controlled and sensitive to recipient design, so the analyst must ask what the assistant's previous turn made relevant before asking what the user meant. Schegloff (2007) supplies the reason a single-turn code cannot recover an action: a turn of a particular kind projects a relevant next, and the next turn is heard against the one it answers, so "no" refusing an offer and "no" reporting an absent fact are different actions under one code. Schegloff, Jefferson and Sacks (1977) show why the correlation is uninterpretable as stated, since repair is where participants locate trouble in the sequence; if the assistant's clarifying question occupies the place where the user's own repair would have occurred, the system has removed the evidence of where the trouble lay and the analyst is left with the demand itself. Garfinkel (1967) and Heritage (1984) make the deeper premise explicit, that sense is settled in the setting and that context is produced and made accountable in the talk rather than acting on it from outside. Perakyla (2004) then tells the team what the reliability coefficient does not do: validity in research on naturally occurring interaction is internal to its logic, and the criteria are transparency of analytic claims, the next-turn proof procedure, in which a speaker's next turn displays their understanding of the preceding one, and the analysis of deviant cases. Stivers (2015) supplies the test to apply to the coding itself, a distinction between coding grounded in sequential observation and coding imported from a variable the analyst brought, and warns that formal coding risks a massive reduction and flattening of complex conduct into simplistic codes. What has to change is therefore the unit of analysis. Code sequences rather than turns, derive each category from observed cases that the team can display, include the assistant's turn as part of the analysed exchange, and report the deviant cases that do not fit, since a study that reports none has either hidden them or not looked. The setting is not neutral either: Drew and Heritage (1992) and Heritage and Clayman (2010) show that institutional interaction has its own constraints and inferential frameworks, so a service encounter is a describable order rather than a baseline. What would strengthen the work is an audit of a sample of exchanges against the deviant-case criterion, a stated derivation for each code with transcribed instances, and a report of what the statistic preserved and what it discarded. ten Have (2007) and Hutchby and Wooffitt (2008) give the practical route: recorded, replayable evidence anchored to segments a reader can inspect, with single case analysis where the corpus will not carry a general claim. Antaki (2011) shows where such work can lead if it is done this way, since applied studies intervene in a practice and can show the sequence they changed rather than reporting a distribution about it. Sidnell and Stivers (2013) is the reference to consult for the machinery before recoding.

The conclusion about gaze and gesture is not supported by the data, because the study never collected the record in which gaze and gesture occur. That is the first assumption to name: that talk is the medium of coordination and that the body is context. Jordan and Henderson (1995) treat talk, non-verbal interaction and the use of artefacts as one organisation, with video as the primary record and replay as the instrument, so a design that leaves the hands and eyes unrecorded has removed the evidence for the claim it makes. Heath, Hindmarsh and Luff (2010) put the point in practical form: the record is collected before the analytic question settles, and what is not collected cannot be recovered later. Second, the study assumes that relevance can be assigned in advance by the analyst. Goodwin (2000) defines the contextual configuration as the array of fields participants demonstrably orient to, expressly not a set the analyst imposes to code context, and Hindmarsh and Heath (2000) show that even the referent of a deictic expression is fixed by gaze, gesture and bodily orientation, which talk alone leaves indeterminate. A claim that gaze played little part therefore cannot be read off a transcript of a shared screen. Third, the codes assume that each turn carries its meaning as content. Goodwin (1994) shows that coding schemes, highlighting and graphic representations are practices that shape what a profession sees, so a code is work rather than a neutral report, and Mondada (2018) argues that transcription choices have analytic and conceptual consequences, since what a system cannot represent, such as body arrangements in interactional spaces or camera movement, cannot be claimed from it. Mondada (2016) adds the conceptual consequence: language is one modality among others, and a logo-centric method will find talk dominant because it looked only there. What has to change: record the working session on video, return to the moving record for any claim about gaze, gesture or the manipulation of the images, and re-derive the categories from what the participants make visible to one another rather than from a scheme fixed in advance. The talk codes may survive as one layer. What would strengthen the work: report the segment for each claim, ask whether the participants' own next action displays orientation to the mode the coding treats as dominant, as the analyst-led framework of Norris (2004) would require, and state what observation would count against the conclusion. The comparison case is also available, since Streeck, Goodwin and LeBaron (2011) show the same apparatus applied across clinical and other material settings, and Heath and Luff (2000) show video-based studies informing design claims about systems in use. Finally, the team should say which tradition it is working in, because Jewitt (2014) shows that multimodality is a family of approaches rather than one method, and a social-semiotic study such as Bezemer and Kress (2016) would ask a different question about the resources in play, using still and printed material where that fits the claim.

The design makes the planning model's assumption that Suchman (1987) is written against: that the work is the execution of the workflow the managers described, so that a faithful rendering of that plan is a faithful model of the work. The workflow was collected as a requirements list, which is the move Anderson (1994) argues discards ethnography's analytic value, and the interviews were with managers rather than with the people doing the work. That the staff keep a parallel paper list is the finding Sellen and Harper (2002) would lead you to expect: paper affords annotation, flexible movement between documents and a visible record of progress that the system did not reproduce, so a design built on a model of the document missed the work the document was doing. The analysis that would have caught it is a study of the practice as it is actually conducted. Hutchins (1995) directs that study at the whole system, asking where the knowing sits and what changes when one component is replaced; Lave (1988) warns that the competence in the parallel list is a relation between activity and setting rather than a habit to be trained away; and Bannon (1991) requires the staff to be actors in the design rather than users whose needs are collected once. Dourish and Button (1998) set the standard for judging the repair: the study must change what the design treats as the work, not merely supply extra features, and Dourish (2001) grounds that change in practice rather than in a better model of the user. To strengthen the claim, trace one situated finding into a particular design decision, state in advance what the deployed system will break, for example that a single shared worklist cannot carry the private annotations the staff rely on, and check that prediction against the deployment. Crabtree, Rouncefield and Tolmie (2012) and Randall, Harper and Rouncefield (2007) show what such a trace looks like, and Kuutti and Bannon (2014) with Suchman and colleagues (1999) place the whole question in the practice paradigm, where a technology is assessed in relation to the sites of its production and use.

The list establishes that certain procedures were performed, not that the interpretation is warranted, which is Barbour's (2001) point that technical fixes such as multiple coding and respondent validation confer no rigour unless they are embedded in the analysis. The paper cannot show what the reader needs, because the account of the analysis is thin, and this is the consequence Morse and colleagues (2002) describe when responsibility for rigour moves from the investigator's conduct to the reader's judgement. The reflexive statement is the clearest example: Finlay (2002) distinguishes five variants of reflexive practice, so a sentence that does not name the variant, and what it changed, is not inspectable. Member checking cannot repair this, because Smith and McGannon (2018) conclude that it is ineffective for verification or trustworthiness, and Varpio and colleagues (2017) show how the term became expected without its implications being thought through. Ask instead for the analytic decisions and the counter-cases. Morse (2015) lists negative case analysis among the strategies that carry weight, and the two interviews that fit the tool well are exactly such cases, so the authors should say how the reading of distrust survived them. Frame the request the way Levitt and colleagues (2017) frame review, as a question about fidelity to the subject matter and utility in answering the research question, and the study's claim will be either strengthened or withdrawn on evidence the reader can see.

Measure time to complete a save task and the error rate, and record them separately, because speed and accuracy can move in opposite directions. Treat the construct as efficient and reliable saving rather than as preference, and add a brief subjective rating only as a secondary measure. Use a within-subjects design so that between-person variability leaves the error term, and counterbalance the order so that half the participants meet the shortcut first. That balances practice and carry-over without removing them, which matters because a shortcut learned first may contaminate the later toolbar trial. Justify the sample size before collecting: state the smallest difference in save time that would change a decision, estimate the variability of save times from a pilot, and compute the number of participants needed for 0.80 power at that difference. Analyse with a paired t test, and check first that the within-participant differences are roughly normal and free of severe outliers; use a Wilcoxon signed-rank test and describe it as a comparison of ranks if that check fails. Report the mean difference, its confidence interval, the individual-participant differences in a figure, and the assumption checks. The design cannot establish how either method performs after weeks of habituation, since a short laboratory task measures early learning, and it cannot speak to users who never learned the shortcut at all.

The first point is that a p value of .04 establishes only that data this extreme would be unlikely if the null were true. It says nothing about the size of the benefit, and two per cent on a click metric may be smaller than the run-to-run variation in the metric itself. Ask for the effect with a confidence interval, and for the interval computed on the primary metric that was prespecified before the test began, not on whichever metric happened to move. The second point is the number of comparisons. If the team inspected a dashboard of metrics, tried several segment cuts, or checked results repeatedly and stopped when the number looked good, the effective false positive rate is far above five per cent; Simmons, Nelson and Simonsohn's simulations put the combined figure at about 61 per cent. Ask for the analysis plan, the stopping rule and the number of metrics examined. The third point is the metric itself. Clicks may be a proxy for value or a proxy for interruption, and the test should show a secondary outcome such as completion or retention moving in the same direction. The fourth is duration: a short test can show a novelty effect that fades. Then the ethical question. Some users received the version the team now believes is worse, and that exposure was not disclosed to them. Ask which metric was optimised, whether it is the one users would choose, and whether the experiment could be described to participants without embarrassment. If it could not, the result is not the only problem.

Begin with the claim rather than the number. Four points on a seven-point agreement scale is a reading from an instrument, and the team has given it the name of a construct, satisfaction, without establishing that the scale measures that construct. Stevens (1946) matters first: a seven-point agreement scale is at best ordinal, so the mean difference assumes equal intervals, and the report should state that assumption rather than let the difference carry a precision the scale does not have. Campbell and Fiske (1959) set the second demand, that a construct needs convergence across at least two methods and discrimination from related traits, so a single questionnaire cannot establish satisfaction, and the fall in completion time is a separate measure whose low correlation with satisfaction is exactly what Hornbaek and Law (2007) report. Hassenzahl (2004) shows why pooling is unsafe, since perceived usability and hedonic attributes both feed satisfaction yet respond to different things and move on different schedules, so the single score may be tracking perceived usability after use rather than satisfaction as such. The team would need a second, independent measure of satisfaction, an argument that the scale's structure matches the construct, and a test of whether the change moved the pragmatic component while leaving the hedonic component untouched, following Flake and Fried (2020) on reporting the measurement decision in the paper. Until then the honest conclusion is that the redesign changed some measured quantities, not that satisfaction improved as a construct.

Separate three claims in the team's sentence. First, alpha of .94 is reported as though it settled the instrument's quality, but Cronbach (1951) showed that alpha is the mean of all split-half reliabilities and therefore a lower bound on reliability, and it presupposes that the items measure one thing rather than demonstrating it. Sijtsma (2009) adds that alpha always takes a value and is unrelated to the internal structure of the test, so the high figure cannot establish unidimensionality. Second, the single-construct claim is contradicted by the evidence, since Lewis and Sauro (2009) factor-analysed two independent SUS datasets and found a usable factor of eight items and a learnable factor of two, both reliable and highly correlated with the total, so the total score averages two constructs. A high alpha coexisting with that structure is exactly what Green, Lissitz and Mulaik (1977) predict, because alpha can be high for heterogeneous items. Third, reading 78 as usable imports the interpretation layer of Bangor, Kortum and Miller (2008) and Kortum and Bangor (2013), whose norms were built on products rated from memory rather than through task use, and reliability is not validity. To defend the claim the team would need the factor structure reported, a reliability estimate that does not assume unidimensionality such as omega total or the greatest lower bound (McNeish, 2018), and evidence that the score relates to what the construct should predict, following the validation sequence set out by Boateng and colleagues (2018).

Separate four claims. First, the counterbalancing: Bradley (1958) shows that a Latin square with an even number of conditions gives each condition each predecessor equally often, so the order effect is distributed evenly across conditions rather than cancelled. The team may report that no condition benefited systematically from its position, and nothing more. Second, the range effect the design cannot reach: Poulton (1973) showed that exposure to a range of values lends contextual comparison to every scenario other than the first, so a treatment's apparent advantage depends on the other layouts offered alongside it. Layout A was judged inside a set of three layouts, so its apparent speed is relative to that set, and the same layout in a different set could be judged differently. Third, the design's sensitivity: the team gained efficiency by removing between-person variance from the error term (Greenwald, 1976), and they accepted practice, sensitisation and carry-over effects they have not estimated (Greenwald, 1976; Charness, Gneezy and Kuhn, 2012). Changing the test does not address this. The failed sphericity check is a fact about the univariate route, and moving to a multivariate or non-parametric analysis is a legitimate response to that assumption (O'Brien and Kaiser, 1985); Maxwell and Delaney (2003) make the wider point that the analysis is derived from the design's structure, so the switch neither repairs the design nor restores the independence the arrangement has given up. Fourth, the generalisation claim: the unit follows Mook (1983) in treating generalisation as a separate question, so the team must say which population and which set of alternatives they intend the claim to cover, and then test it. What would earn the causal claim is a between-subjects randomised version of the same question, since randomisation balances unmeasured variables that post-hoc adjustment cannot reach (Rubin, 1974; Imai, King and Stuart, 2008).

Begin by naming the decision, because the unit's claim is that the sample size is an inference from it. If a redesign ships only when completion rises by five percentage points, then five points is the smallest effect that would change the decision, and the sample size follows from that figure rather than from a medium effect of .50 (Lakens, 2022). Say which justification you are using, since Lakens allows an a priori power analysis, a desired precision or a resource constraint, but requires you to name it. If the study is a two-condition within-participant comparison, more than 50 participants are needed for 80 per cent power at d = .4, and a between-groups factor or an interaction pushes the figure to 100 or 200 (Brysbaert, 2019); if the design has stimuli as well as participants, plan for about 1,600 observations per condition rather than counting people alone (Brysbaert and Stevens, 2018). Then state the errors you will tolerate. Gelman and Carlin (2014) recommend reporting the probability of a wrong sign, type S, and the factor by which a magnitude is exaggerated, type M, because a study can be powered for the effect it assumed and still mislead when the true effect is smaller. That is what fixes what the study can conclude. A significant result at the planned size is an estimate with an interval, not a promise that the effect is as large as observed, since small studies inflate what they find (Button et al., 2013). A null result at sufficient planned power is evidence that the effect is smaller than the threshold you chose, which is a decision-relevant statement, whereas a null from an underpowered study is usually no evidence at all. Finally, scope the claim to studies that estimate an effect. CHI's modal sample of 12 covers interview and observational work for which power is not the criterion (Caine, 2016), and even within quantitative work the problem is concentrated in experiments rather than correlations (Schweizer and Furley, 2016), so the argument bites on the studies that make a statistical effect claim.

Begin by separating what the p value licenses from what the claim asserts, because the claim treats a decision as a magnitude. The p value answers a question about data under a null hypothesis and does not measure the size or importance of the effect (Wasserstein and Lazar, 2016), so the honest rewrite reports the difference in completion rates as an effect with an interval: for example, a rise of four percentage points with a 95 per cent interval from two to six points. That interval does work the verdict cannot, because a reader can see whether the whole interval clears the threshold that would matter for a shipping decision (Cumming, 2013). Say how the interval was computed. A between-subjects comparison of two proportions needs an interval from the difference of independent proportions, whereas a within-subjects comparison needs an interval that accounts for the correlation between conditions, and Dragicevic (2016) warns that the ordinary calculation is wrong in within-subject designs and that unlabelled error bars mislead. State the effect size measure explicitly, since a difference in proportions, a risk ratio and a standardised difference answer different questions (Kelley and Preacher, 2012). Add a caveat from the interval literature: an interval is not a Bayesian statement of plausibility, and readers systematically over-interpret it as the probability that the true value lies inside (Hoekstra et al., 2014; Morey et al., 2015), so the report must define it. Then take a position. The better argument is for supplementation rather than abandonment at the level of an individual study, because the p value carries information about compatibility with a null when the design is prespecified, and because the practice gap Fritz and colleagues (2012) document shows that even the modest step of reporting effect sizes with intervals has not been taken. Abandoning the test while intervals are computed wrongly and read wrongly would trade one misread statistic for another that is misread in a new way. But the abandonment case has force at the level of the literature rather than the single paper: McShane and colleagues (2019) are right that a threshold screening role distorts what gets published, so the field should stop treating p below .05 as the criterion of a discovery, even while an individual prespecified test is reported. That position is closest to Lakens and colleagues (2018), who ask for a justified alpha rather than a new universal convention, and it is against Benjamin and colleagues (2017), whose .005 proposal fixes the threshold instead of questioning it.

The plan begins by naming the estimand, because the unit's argument is that the method choice is a claim about what the study may assert. Task completion time is bounded below and open above, so a mean is a value many participants never produce; the plan should therefore report a trimmed centre rather than the arithmetic mean, and the trimming rule has to be fixed now rather than after the data are seen (Wilcox, 2012). The rule for handling the long idle trials, including how they are classified and whether they are retained, recoded or removed, is part of the plan for the same reason: Leys and colleagues (2019) argue that such a rule must be pre-specified because it affects validity and reproducibility, and a decision made after seeing the tail is exactly the analytic flexibility the rule exists to prevent. With a trimming proportion fixed in advance, conventionally 20 per cent in this literature, the two conditions can be compared with Yuen's trimmed t, which uses Winsorised variances and remains robust to non-normality and to unequal variances (Yuen, 1974). That test licenses a claim about the trimmed centre, not about the mean, and the report must say so. The interval comes from the bootstrap, because a trimmed mean has no textbook standard error and Efron's resampling method estimates its sampling distribution without a parametric assumption (Efron, 1979); the plan should state the number of resamples and report the interval alongside the estimate. The design also has structure that the analysis must respect. Each participant contributes ten trials, and if the flow is tested on a fixed set of screens or items, those items are seen by every participant, so the analysis should use a mixed model with crossed random effects for participants and items rather than a by-subjects analysis, because ignoring item variability present in the data can inflate the Type I error rate (Baayen et al., 2008; Judd et al., 2012). Reporting should give the trimmed centre, its interval, the number of trials trimmed and the exact rule, so a reader can recompute it. If the data proved far more skewed than expected, the honest response is not to change the estimator after the fact; it is to report the pre-specified analysis, and then report the sensitivity of the conclusion to the trimming rule as a labelled secondary analysis, stating clearly which claim each version supports.

The plan begins with the statistic, because the unit's argument is that the choice of statistic decides what claim the study may make. A sequential design that tests after every fourth participant is exactly the practice a p value cannot support, since the p value is computed against a sampling plan that fixes when to stop, so the team should report a Bayes factor, which is a ratio of the probabilities the two models assign to the observed data and does not depend on the intention behind the collection (Dienes, 2011; Wagenmakers et al., 2018). The Bayes factor also lets the team state the conclusion they actually want, evidence for the null as well as against it, which a non-significant p value cannot supply (Rouder et al., 2009; Keysers et al., 2020). Freedom from the stopping rule does not remove the need for a design: the team should still fix a maximum sample size and use Bayes factor design analysis to compute the expected strength of evidence, the expected sample size and the probability of misleading evidence, so that stopping early is a planned property of the design rather than an opportunity taken (Schonbrodt and Wagenmakers, 2018). The prior must then be chosen and justified in advance. Ly, Verhagen and Wagenmakers show that the default tests for the nullity of a normal mean rest on a named default prior, and Rouder and colleagues' default is a Cauchy distribution whose tails are so heavy that neither its mean nor its variance exists, so the plan should state which default it uses and why it fits the expected size of the shortcut's effect (Rouder et al., 2009; Ly et al., 2016). Guarding against a conclusion that depends on that prior is the next requirement, and it is what separates a defensible Bayesian analysis from a chosen number. The plan should include a sensitivity analysis across the priors reasonable analysts would choose, and prior and posterior predictive checks that test whether the model captures the data, which is the principled workflow Schad, Betancourt and Vasishth set out (Schad et al., 2021). Reporting should follow van Doorn and colleagues: give the Bayes factor with the prior settings used, justify those settings, and discuss the robustness of the result, so that a reader who prefers a different prior can recompute and judge it (van Doorn et al., 2021). If the evidence were still inconclusive at the maximum sample size, the plan should report that as an uninformative result rather than as evidence for the null, because the middle band supports neither hypothesis, and the honest addition would be a stated interval for the effect size and a decision about whether the study is worth extending (Dienes and McLatchie, 2018).

The plan begins by treating the 12 per cent gain as an upper estimate rather than a fact, because replication programmes report effects about half to a quarter of the size of the originals, and the Reproducibility Project: Cancer Biology found median replication effects 85 per cent smaller than the originals (Open Science Collaboration, 2015; Klein et al., 2018; Errington et al., 2021). The team should then state, in a registration written before any further data collection, the primary outcome, the stopping rule, the covariates and the set of conditions it will compare, since those are the researcher degrees of freedom that Simmons, Nelson and Simonsohn showed can manufacture a significant result and that Wicherts and colleagues listed as a checklist (Simmons, Nelson and Simonsohn, 2011; Wicherts et al., 2016). The registration should also fix the smallest completion gain that would justify shipping, so that practical significance is decided in advance rather than argued after the fact. The report should give the effect with a confidence interval and state that the finding has not been independently replicated in HCI, since Hornbaek and colleagues found only about 3 per cent of 891 HCI papers attempting replication, which leaves the field's own replication rate unmeasured (Hornbaek et al., 2014). To make the claim about HCI rather than about one laboratory, the team needs a preregistered, adequately powered replication using the original materials, of the kind Many Labs 2 ran across many samples (Klein et al., 2018). If that study returns an interval compatible with no effect, the honest conclusion is that the result is uninformative rather than evidence that the menu does not help, and the team should say so plainly rather than claiming a null effect.

The plan starts with the metric, because at platform scale the metric is what the test resolves: many ideas move key metrics by around one per cent and are not well estimated beforehand, so the team should register an overall evaluation criterion and the counter-metrics expected to move against it before the variant is built, and use the trustworthiness checks the platform literature describes rather than one headline figure (Kohavi et al., 2013; Kohavi, Tang and Xu, 2020). Episodes per session is a proxy that can rise while the construct moves the other way, so the registration should name a counter-metric such as sessions ending within seconds or cancellations in the following week, which is what makes a proxy moving against the construct visible. The assumption the test most plausibly breaks is independence: if the service has any sharing or social feature, one user's behaviour can depend on others' assignments, and Eckles, Karrer and Ugander show that standard designs assume the absence of interference and that graph cluster randomisation and neighbour-treatment analyses reduce the resulting bias (Eckles et al., 2016). The team should therefore test the assumption rather than assert it, using the design Saveski and colleagues deployed, which runs a completely randomised and a cluster-based randomised experiment simultaneously and compares the estimates, and which found network effects and bias in standard A/B testing in production (Saveski et al., 2017). Watching the dashboard daily is a mistake, because traditional p values and intervals are unreliable under continuous monitoring; the team should either fix the analysis date in advance or use always valid p values and confidence intervals, which are robust to monitoring and were deployed by the platform Optimizely (Johari et al., 2017). The retention analysis is biased as proposed, since counting users who have not yet cancelled as failures biases the survival estimate downwards; Kaplan and Meier derive the product-limit estimator for the case where the event is missed for some units, and they say the assumption that the lifetime is independent of the loss time deserves careful scrutiny, which is exactly what a heavy user observed for longer violates (Kaplan and Meier, 1958). The team should report the survival curve with an explicit statement of the censoring assumption, and model who abandons with a discrete-time hazard model of the kind Singer and Willett fit using ordinary logistic regression (Singer and Willett, 1993). I would withdraw the claim that the change is good for users if the counter-metric moved against users, if the interference test showed the estimate was biased by spillover, or if the interval on episodes per session was compatible with no effect: a 6 per cent gain in a proxy with an opposite movement in an abandonment metric is evidence that the platform changed what it measures, not that users are better off.

The plan starts with the metric, because an approval rate is an aggregate over a population the council already serves unequally, and the unit's worked case shows what such an aggregate hides: Buolamwini and Gebru found two benchmarks overwhelmingly lighter-skinned and reported errors up to 34.7 per cent for darker-skinned women against a maximum of 0.8 per cent for lighter-skinned men, so the average reported the systems as close to working (Buolamwini and Gebru, 2018). The team should therefore report the approval and refusal rates separately for the groups the policy is meant to reach, including the rate at which each group is wrongly refused, because those two errors fall on different people. The choice of a fairness definition should then be stated openly rather than assumed: Kleinberg, Mullainathan and Raghavan prove that three central fairness conditions cannot be satisfied at once except in highly constrained special cases, and Corbett-Davies and colleagues show that enforcing a popular definition typically produces strongly Pareto dominated policies that can harm the group the definition was meant to protect (Kleinberg et al., 2016; Corbett-Davies et al., 2018). The team should document which definition it chose, who chose it, and what the alternatives would have done to each group. The provenance of the training data is the next problem, since five years of awarded applications records the decisions of an earlier policy rather than the need for insulation; Hutchinson and colleagues propose documenting the decisions behind a dataset at each stage of its construction so that a reader can see what was chosen and contest it (Hutchinson et al., 2021). Finally, the question of who is measured and who benefits belongs at the start rather than the end: Friedman, Kahn and Borning treat human values as requirements to be investigated with the people affected rather than assumed by the designer, and D'Ignazio and Klein ask who benefits and who is excluded from a dataset (Friedman et al., 2008; D'Ignazio and Klein, 2020). I would withdraw the claim that the system is fair if the refusal rate rose for the group whose applications the old policy missed, if the improvement in the aggregate was produced by a subgroup that was already well served, or if the definition of fairness was chosen after the results were seen: any of those would show that the metric was optimised rather than the outcome.

A strong answer names the page and the specific criterion for each failure, because the marking rewards precision over volume. Typical AA failures on public services are a custom widget built from div elements with no role or keyboard behaviour, which fails 4.1.2 Name, Role or Value; low-contrast status text or a pale input border at under 3:1, which fails 1.4.11 Non-text Contrast; a focus outline removed or a sticky banner that hides the focused control, which fails 2.4.7 or 2.4.11; a login with a time-limited one-time code and pasting blocked, which fails 3.3.8 Accessible Authentication (Minimum); and small, tightly packed targets, which fail 2.5.8 Target Size (Minimum). Prioritisation should follow user impact rather than the order of the guidelines. A failure that blocks the whole task, such as a keyboard trap or an unreachable submit button, outranks a cosmetic one, because the person cannot complete the goal at all. The evidence section separates a good answer from a fluent one. For each claim, name how it could be shown: a keyboard-only walkthrough recording where focus is lost; a screen reader transcript showing the missing name and role; a measured contrast ratio from the computed colours; a task attempt with a person who uses assistive technology. An audit that asserts confusion without naming how confusion could be demonstrated is criticism, and the marking treats the two differently.

Ability-based design starts from what a person can do rather than from a diagnosis. The brief should therefore be reframed as a measurement question: what pointing accuracy, target dwell time and reach does the task actually demand, and how do those demands vary across the two situations named? An accommodation approach would keep the interface unchanged and issue an add-on, such as a stylus or a screen magnifier, that the user must acquire and configure. Ability-based design instead changes the system. Following Wobbrock and colleagues, the team would instrument the app to observe target acquisition, then adapt the interface by enlarging targets, relaxing the drag gestures that assume steady pointing, and offering a stable set of controls within thumb reach. The argument for preferring this is that it moves the burden of conforming from the human to the software, and it benefits the one-handed commuter as much as the person with a tremor. The honest limits matter for the marks. Measuring and modelling ability is difficult and noisy; adaptive changes can be disorienting if they are not predictable; and the paper describes research prototypes rather than a shipped, validated product. At the end of the work the team could claim that a specific measured task became easier for the participants tested, with the method and sample stated. It could not claim that the app is now accessible to everyone, because that would exceed what any small evaluation can support.

Separate three claims. First, the removal of barriers is the social model working as intended. UPIAS (1976) defined disability as something imposed on top of impairment by an isolating society, and Oliver (1990) made that into a political account in which exclusion is done to people by their environments; on that basis enlarged targets, alt text and contrast are exactly the right remedies, and the brief is right that disability is not a private defect. Second, the sentence 'nothing left in the user to accommodate' is where the unit argues the model over-reaches. Shakespeare and Watson (2002) object that the strong social model cannot account for the experience of impairment and that the impairment and disability dualism is unstable, and Anastasiou and Kauffman (2013) add that reducing disability to a single social dimension downplays biological and mental conditions and can harm people by denying their needs; Hviding (2013) questions whether the tradition's normative claims can carry that weight. The team has treated a tool of political analysis as a complete account of experience. Third, the relational position the unit says the field actually runs on is stated by Thomas (2004), who wants the social relational understanding revived, and by Shakespeare (2006), who treats disability as the product of the interaction between a body and a society with both treated as real; Siebers (2008) reaches it through complex embodiment and Kafer (2013) through a political and relational model. On that view a tremor that makes a target hard to hit and chronic pain that ends a session are not barriers the team has removed but properties of the situation still to be designed for. To show the design meets the needs of users with significant impairments the team would need evidence of consequences, which the unit argues the literature lacks: following a single service through both framings and measuring whether the most significantly impaired users fare as well as they would under a relational approach, using the instruments Bickenbach and colleagues (1999) describe, since those instruments encode a framework and can be compared. Until then the brief has asserted a conclusion rather than tested it.

Separate four claims. First, the ambition is the right one and it is the founding definition: Connell and colleagues (1997) define universal design as usable by all people to the greatest extent possible without adaptation or specialised design, and Mace (1998) shows the promise doing real work in a domain that accessibility codes do not reach. Proactive design of a booking service is therefore a better aim than repairing barriers after disabled patients complain. Second, the sentence 'our own design team has the expertise to judge that' is where the unit locates the problem. Newell and Gregor (2000) set out user sensitive inclusive design, in which disabled people are involved throughout but the research questions and methods stay with trained researchers, so the designer settles the target in advance and the user refines it; and Iwarsson and Stahl (2003) explain that accessibility is a relation between a person and an environment, so no service is accessible in itself. A claim to work for every patient is a coverage claim, and coverage cannot be certified from inside the design team; the one published test of such a claim, a universally designed hotel judged by 34 guests including seven wheelchair users, still found heavy doors, high beds and too little turning space (Vahedi, Quinn and Cavuoto, 2025). Third, the phrase 'we will not need individual requests' is exactly the assumption Keates and Clarkson (2004) found unfulfilled in practice, since they record very limited industrial uptake and continuing unnecessary exclusion, and Imrie (2011) argues the concept has become orthodoxy without evaluation of its theoretical content, with marketisation threatening to reduce a right of access to a market transaction. Fourth, the literature disagrees about whether the promise can be kept. The renewal position, Steinfeld and Maisel (2012), Heylighen and colleagues (2017) and Lid (2014), holds that universal design is a maturing practice attached to a relational model of disability such as the Convention on the Rights of Persons with Disabilities presupposes, and that the failure is one of implementation. The critical position, Imrie (2011) with Keates and Clarkson (2004), holds that the concept is unevaluated and that its vocabulary does reassurance work. To accept the brief I would require measured coverage: who among the trust's actual patients can complete a booking and who cannot, tested with people who use assistive technology, reported by population rather than by feature count. WCAG conformance cannot supply that evidence, and note also that the naming is not settled, since Persson and colleagues (2015) find little consensus on the terms while Clarkson and Coleman (2015) treat them as one approach under national conventions, so the trust should say which requirement a contract will hold it to.

Begin by separating the two kinds of adaptation, because the sides are comparing different things. The engineers' evidence is Gajos, Wobbrock and Weld (2008), where participants with motor impairments using interfaces generated by SUPPLE++ were 26.4 per cent faster and made 73 per cent fewer errors, and Gajos, Weld and Wobbrock (2010), who show that generation is computationally feasible at run time. That evidence is strong but narrow: it compares generated interfaces with manufacturers' defaults rather than with a user-directed alternative, and it rests on a one-time ability model rather than on validated measurement. The consultant's evidence is Findlater and McGrenere (2004), who found a static menu significantly faster than an adaptive one and a user-arranged menu faster than the adaptive one under certain conditions, and Lavie and Meyer (2010), who found that adaptation helped in familiar, routine situations but raised cognitive workload when the situation was unfamiliar. Hurst and Tobias (2011) add that abandonment of purchased assistive technology runs at 35 per cent or more, and that control over design decisions motivated users to build their own devices. A study that settles the dispute needs the same users, tasks and interface under automatic adaptation and under user-directed adaptation with transparency and reversal, run over weeks of ordinary use, reporting throughput, errors and abandonment together with the accuracy of the ability model against the user's measured performance. Mott and colleagues (2018) show the field-study half of that design, interviewing people about real practice rather than about an experimenter's task. Mitchell and Wobbrock (2024) supply the reason the model's accuracy must be reported: ability has not been operationalised consistently, so a gain attributed to adaptation may come from an unvalidated measure. The band is the comparison. A system-authority result requires automatic adaptation to match or beat user-directed adaptation on performance and abandonment across both routine and non-routine use; a user-authority result requires the automatic condition to lose on abandonment or on performance when the situation is unfamiliar.

Begin by separating the two claims the statement runs together. A conformance claim is a claim about a resource against a set of testable criteria; it is not a claim that a person can achieve a goal. Petrie and colleagues (2015) analysed 50 definitions of web accessibility and proposed one centred on people being able to use websites across contexts of use with mainstream and assistive technologies, which is a goal-centred definition rather than a property a checker confirms. On that definition the body has not shown what it asserts. The evidence behind the claim is weak for the same reason. An automated scan can test only the criteria a tool can decide, and the standard itself describes accessibility as testable with a combination of automated testing and human evaluation (W3C, 2023), so a clean scan is a partial result presented as a complete one. Brajnik (2008) explains why the two methods are not interchangeable: conformance testing and user-based testing answer different questions and detect different problems. The panel's report is the user-based half the body omitted. The body could defend itself by narrowing its claim, and the arguments are real. It can say that conformance is what procurement and policy demand, that a verifiable floor is better than an unverifiable aspiration, and that Chisholm and Henry (2005) show access depends on several components, so a failure in the content or in the assistive technology is not necessarily a failure of its content. Caldwell and colleagues (2008) and Reid and Snow-Weaver (2008) support the position that testable criteria are what made the standard enforceable at all. What settles the question is evidence the statement did not supply: task-based evaluation with screen reader users on the form, recording whether they completed the task, where they stopped and which of the failures fall inside the criteria. Power and colleagues (2012) provide the design and the baseline, having logged 1,383 user problems with 32 blind users and found only 50.4 per cent covered by WCAG 2.0 success criteria, with 16.7 per cent of websites implementing a recommended technique that did not solve the problem. If the panel's failures fall inside the criteria, the scan was inadequate; if they fall outside, the standard was, which is the outcome Lewthwaite (2014) predicts when universal and automated approaches are applied irrespective of context. A defensible statement should therefore report what was tested, how, and by whom, and should not translate conformance into access.

Separate the three claims the sentence runs together: that the page conforms, that the scan was clean, and that disabled users can complete the checkout. Only the first is what a conformance result establishes, and even that only for the criteria a defined scope tests. The scan establishes nothing about the user, because Vigo, Brown and Conway (2013) benchmarked six tools and found coverage of at most 50 per cent of success criteria, so half the criteria were never examined, and of those examined only about four in ten were caught, with correctness between 66 and 71 per cent. A clean scan therefore reports the absence of the failures a tool can detect, not the absence of failures. The method is also a choice with consequences: Brajnik (2006) found that a barrier walkthrough improved correctness by 9 to 60 per cent over a standards review, so the method shapes the verdict, and Brajnik, Yesilada and Harper (2011) found that expertise changes validity and reliability and that more judges produce more diverse outputs rather than converging. The team's single automated pass sits at the narrow end of both variables. The strongest defence is that conformance is the scalable floor and that something must run on every page; Hanson and Richards (2013) measured fourteen years of improvement across more than a hundred sites with automated analysis, and Campoverde-Molina and colleagues (2020) found that twenty of twenty-five studies of educational sites used tools alone, so the team is using the field's standard method. The defence narrows the claim, though, rather than supporting it. What would settle the question is the missing study: a named tool, its coverage and completeness reported on the checkout scope, crossed with task-based outcomes for disabled users on the same page, counting the tool's false negatives as named user failures. Romen and Svanaes (2012) provide the user-side outcome, since compliant pages still posed barriers, and Petrie and Kheir (2007) supply the severity measure, having found no significant agreement between guideline priority ratings and the problems participants actually rated severe.

The laboratory trial establishes that the device met its specification under laboratory conditions. It does not establish that the device fitted the users' activities, that it reached them through a service able to support it, or that it changed anything they cared about. The unit's first move is to treat the device as one element in a system: Cook and Polgar (2015) place the technology inside an activity performed by a person in a context, and Fuhrer and colleagues (2003) make outcomes result from interaction among device type, user and environment, with procurement and an introductory period of use in the chain. The trial holds two of those variables constant, so it cannot speak to the rest. The service's figure of 48 from 120 is 40 per cent, which is the same order as the 29.3 per cent of devices Phillips and Zhao (1993) found completely abandoned in a sample of 227 adults, where the predictors led by lack of consideration of the user's opinion in selection, easy procurement, poor device performance, and changed need. The service's word, non-adherence, is the weakest part of its account. Jahan, Guitard and Jutai (2024) screened 1,029 articles on failure to use prescribed assistive devices, retained 27, and found the terms used inconsistently and sometimes interchangeably, with only 10 of the 27 defining the term used; a term the field has not standardised cannot carry a conclusion about willingness. The strongest defence is that the service is reporting a real signal with the vocabulary available to it, and that continued use matters: Riemer-Reiss (1999) found relative advantage and consumer involvement significantly predicted discontinuance, and Scherer (2002) built the Matching Person and Technology approach on the claim that use follows from the relationship among person, technology and environment. That defence supports better matching rather than blame, because both results put the leverage in selection and involvement. Two further findings bound the claim. Provision is a real constraint, since de Witte and colleagues (2018) set out the elements a service must supply, from affordable products and advice to training, follow-up and repair, and Borg, Lindstrom and Larsson (2009) found that in low- and middle-income countries only about 5 to 15 per cent of people who need assistive technology have access at all, a figure the unit also sets against the World Health Organization and UNICEF (2022), whose first global estimate puts need above 2.5 billion people with nearly a billion denied access. Non-use is also not self-evidently failure, since Scherer and Glueckauf (2005) locate the value of the technology in activities and participation, and Shinohara and Wobbrock (2011) found that functional access took priority over self-consciousness, so a user's decision to stop can be a judgement about goals rather than a deficit. The study that would settle the matter would take one device class, state its operational definition of non-use, record the fit and provision measures before supply, and measure participation afterwards; Jahan and colleagues (2024) supply the definition, Scherer and Glueckauf (2005) the outcome, and de Witte and colleagues (2018) the provision variables. Without those three, the 40 per cent is a fact about the service's record-keeping as much as about its clients.

The visual review and the automated scan establish that the page renders as intended and that machine-checkable criteria are met. They cannot establish that the accessibility tree carries what a user needs, because both inspect the rendered page rather than the interaction model a screen reader reads. Borodin and colleagues (2010) show that screen reader browsing strategies differ sharply from sighted browsing, and WebAIM (2024) found that 71.6 per cent of respondents navigate through headings first on a lengthy page and 88.8 per cent find heading levels useful, so heading structure is the route rather than decoration. The first failure, results as one unlabelled region, collapses that route into a single node. The second, headings restyled as paragraphs, does the same damage invisibly, and no scan flags a heading that was never marked up. The third, linked cover images with no alternative text, is the barrier Bigham and colleagues (2006) documented when they found a large fraction of significant images carried no alternative text and built WebInSight to supply labels. The lead's proposal to add ARIA roles is the mistake Faulkner and MacDonald (2018) warn against first: a role added to an element that did not need one does not restore semantics but makes the tree report something false, and a larger scan repeats the original error. The remedy should be structural: mark the results as a list with real headings and give the images described labels, as Srinivasan and colleagues (2023) did by making data navigable structure rather than a text stream. The study that would settle the question closes one defect class and measures screen reader users' task performance and time on the fixed pages against matched pages that were never fixed, using the time and frustration measures of Lazar and colleagues (2007) and the structure measures of Srinivasan and colleagues (2023), with the sampling frame of Mack and colleagues (2021).

The two remedies correspond to the two positions the unit sets out. The capability position, which Charness and Boot (2009) represent most strongly, locates difficulty in the person and responds with training and support, so the first remedy fits it. The relational position holds that the difficulty is produced at the interface: Milton (2012) argues that the mismatch between autistic and non-autistic people is mutual, Dalton (2013) argues that design carries unexamined assumptions about cognitive processing, and Kapp (2020) and Gillespie-Lynch and colleagues (2017) establish that autistic adults hold the expertise that design currently excludes. Zolyomi and colleagues (2019) give the mechanism in the exact setting: their interviewees reported sensory sensitivities, cognitive load and anxiety shaping channel preference, and coped by moderating sensory input and by masking as non-autistic behaviour, with more stress reported when those strategies failed. The staff network's report is that pattern, which means the training module asks the users to pay more of the cost they are already paying. The platform changes should therefore be recommended, because they reduce the demand the system makes rather than the user's capacity to meet it. What they cannot establish on their own is that access improved: agenda-setting, audio-only joining and notification control are design intentions, and the unit's warning about guidance applies to them, since the W3C (2021) objectives are non-normative and nobody is obliged to test them. The study that would settle it takes those changes as the intervention, measures meeting participation, reported exhaustion and the coping work users perform against matched teams left on the default settings, records the demand the platform imposes separately from the capability of the user, and is designed with autistic staff rather than for them.

On Arnstein's (1969) ladder the workshop sits low: participants were shown finished options and asked to choose, which is consultation or placation rather than partnership, because the problem, the options and the criteria were all set by the team. The unit's reviews support the reviewer. Benton and Johnson (2015) found that involvement of children with special educational needs and disabilities was broad in role but concentrated where it shapes the artefact least, and Borjesson and colleagues (2015) found that most developmentally diverse children took a passive, observational role across the requirements, design and evaluation phases while one group was most often actively involved. The library's design repeats that pattern in another setting: presence without authority. Oliver (1992) and Charlton (1998) supply the standard against which the claim is measured, since both locate the transfer of authority in the social relations of research rather than in the interview technique, and Walmsley (2001) separates inclusion from authority, which is the distinction the project has collapsed. The positions in the contested debate divide as follows. Supporting the reviewer: Benton and Johnson (2015) and Borjesson and colleagues (2015) record involvement thin in authority, and Barnes (2003) reports uneven progress from inside the emancipatory movement. Opposing the reviewer: Arnstein (1969) allows that the higher rungs exist, Bjorgvinsson and colleagues (2012) show agonistic participatory design working where users hold real disagreement, and Frauenberger, Good and Keay-Bright (2011) describe a case in which children's perspectives shaped the artefact. Bennett and Rosner (2019) sharpen the challenge, since a single preference exercise risks aligning the team with an imagined user rather than ceding authority to the real one. What would settle it is not a further opinion but a record of who set the research question, who held a veto over the design, and who controlled the findings, kept over the life of the project; Benton and Johnson (2015) supply the variables to classify the roles held, and Frauenberger and colleagues (2015) supply the standard of accountability, which requires participatory choices to be open to debate and critique. The study that would show whether access improved takes disabled students as participants from the start, including some who use assistive technology, sets the research question with them, and compares the redesign against the previous interface on task success, time and errors for matched users, recording the interface's demands separately from the user's capability. It should also report, using the accountability language of Frauenberger and colleagues (2015), who could have rejected each design decision.

The failure is consistent with this unit's thesis that the procurement contract and the timing of the duty, not the statute alone, decide what is built. Olalere and Lazar (2011) found that federal sites failed an explicit legal duty while their accessibility statements often gave no useful information, and this case repeats that pattern, since a self-declared statement has stood in for measured access. Ferri and Favalli (2018) found that harmonised standards and enforcement lagged the rights the framework promised, which describes a contract that named a standard but never tested against users. The failure supports the implementation position: the duty existed and the platform still failed. It does not on its own support the statute-only position of Wentz, Jaeger and Lazar (2011), because their argument concerns United States law and this is a different jurisdiction, though their account of retrofitting fits the twelve-month gap between launch and the reported failure, in which the fix arrives after users have already been excluded. It also tests the rights position of Lazar and Stein (2017), since a right without a remedy does not reach the patient. What would settle the disagreement is the measurement the unit calls for, using the design of the Disability Rights Commission (2004), which combined an audit of 1,000 home pages with task success rates for disabled users at a constant standard; applied before and after the launch, it would separate the effect of the legal duty from the effect of the contract terms. The procurement terms that would have changed the outcome are the levers the unit identifies: name the technical standard, here EN 301 549 with WCAG for web content (ETSI, 2021), require conformance testing with assistive-technology users, and write the test results and a remedy into the contract before award rather than the statement published after launch.

A strong answer fixes the task before it fixes the measures. Name the flow, the entry point and the definition of completion, because task success is only meaningful against a stated goal. Then choose the small set of standard measures: completion, time on task, error count, and a self-report. Say how each is operationalised. For error count, take the lesson from text entry measurement and decide in advance whether you count corrected errors, uncorrected errors or both, since a participant who leaves an error in place and one who repairs it are different findings, and a single number hides the choice. For time, state whether you are reporting the session as run or a silent condition, because relaxed thinking aloud lengthens tasks and widens visual attention, so timings collected that way are not comparable with unmoderated data. Either run the comparison without verbalisation and collect reasoning afterwards, or keep the timing and label it honestly. Use the System Usability Scale as an attitudinal summary and report the value with its interpretation, remembering that the score is not a percentage. Add a workload measure separately, because a checkout that is easy but exhausting is a different problem from one that is difficult. Write the report in three separated parts: observation with the participant count beside each finding, inference clearly marked, and recommendation with the smallest change that would test it. The design of the study is itself the interest rate the project pays.

The evidence offered has two separate problems, and they should be separated. The first is about the method. A heuristic review predicts problems rather than measuring behaviour, so it can speak to whether users will find the new interaction, but it cannot tell the team what users will actually do or how long it will take them. The second is about reliability. Hertzum and Jacobsen found that two evaluators using the same method agree on only a fraction of the problems they report, and that the effect extends to severity judgement, so one consultant's list and its ranking should not carry a decision of this size. The study I would run is a user-defined gesture study in the style of Wobbrock, Morris and Wilson: show participants the effect of each command and ask them to perform the cause, then compute agreement per command. Commands with a shared gesture can become gesture accelerators. Commands with weak agreement should keep a visible control, because a hidden gesture imposes a discoverability cost on every new user. I would add a small measured comparison against the existing toolbar, using completion, time and error count with the same participants and counterbalanced order, and I would treat the gesture arm as better only if it is not worse on errors. I would also state the participant numbers and the fact that the sample supports iteration rather than a population claim.

Begin by making the task computable, because the whole argument turns on it. Name the amplitude as the distance from the resting position of the pointing device or thumb to the centre of the target, and the width as the extent of the target along the axis of approach. Say how you would treat errors: if you use the nominal width you are assuming the four per cent error condition that MacKenzie (1992) shows is needed for the target to behave like noise, so a task with a different error rate should be reported with an effective width taken from the spread of the endpoints, and a target you would describe as a word or a tab should be handled with the two-dimensional reading of width that MacKenzie and Buxton (1992) propose, since a one-dimensional index for a rectangular target can come out unrealistically low or even negative. Say also that this is what the law was built to do: Fitts (1954) found movement time rising linearly with difficulty in reciprocal tapping, and Card, English and Burr (1978) showed that pointing with a mouse is accounted for by the same relation, close to the minimum achievable, which is why a device can be ranked before a single user is recruited. Now take a position on what your figure reports. If you hold the configuration fixed, report the throughput with it and compare two devices on the same task, you are making a relative claim, and the convergence of five published mouse models within 1.2 bits per second that Soukoreff and MacKenzie (2004) report is good evidence that such claims are reproducible. If you then carry the number to a different distance, a different target shape or a different device, you are making an absolute claim that Zhai (2004) argues the measure cannot support, because difficulty divided by movement time moves with the range of difficulty values you chose and equals a true constant only when the regression intercept is zero. The strongest way to write this up is to report the regression as well: slope, intercept, error character, and the difficulty range, which is exactly what Zhai asks for and what Soukoreff and MacKenzie's own recommendations support. Treat Balakrishnan's (2004) survey as the warning about a shortcut: a technique that shrinks distances or grows targets looks as though it beats the constraint, and it often does help for a single isolated target, but the gains largely reconstruct the configuration the law describes and many such techniques do not scale when targets sit close together. Close by naming the comparison that would settle the question for your chosen task: measure the same participants on the same device across at least two difficulty ranges or two target geometries, report throughput beside slope, intercept and error character for each, and show whether the ranking of two devices changes. If it does not change, your figure is portable within the class of configurations you tested; if it does, the figure was always a statement about the configuration.

Begin by fixing the stimulus, because everything that follows depends on it. Use the 500-phrase set of MacKenzie and Soukoreff (2003) rather than text of your own invention: their argument is that a shared pool is what makes results generalisable, and without it your numbers cannot be compared with the literature you want to join. Then fix the correction policy, and say so in the method, because a participant who repairs every error and one who presses on are performing different tasks. Soukoreff and MacKenzie (2003) show the mechanism: repairs add backspaces, which raises keystrokes per character, and leave a transcript close to the target, which lowers the minimum string distance error rate, while a participant who presses on does the reverse. If you impose one policy and report one error rate, you have measured the policy as much as the keyboard. So instruct participants explicitly, or measure the policy instead of imposing it, and report the corrected and uncorrected error rates separately with correction efficiency, as that framework provides. State the effort term properly as well. MacKenzie (2002) computes keystrokes per character a priori, from the technique and the language, with values for English ranging from about 10 for cursor-key selection to about 0.5 for word prediction, and that range is why the measure is a characteristic rather than a verdict: a predictive keyboard reaches a low value by moving effort into attention, and Quinn and Zhai (2016) found that assertive suggestions cut keyboard actions and were preferred while average time got worse, a result Palin and colleagues (2019) echo in the negative correlation between word prediction use and mobile entry rate. Now pick a technique and make the trade visible. SHARK (Zhai and Kristensson, 2003) and SHARK2 (Kristensson and Zhai, 2004) write familiar words as single gestures and leave unfamiliar ones to be tapped, so the same technique behaves differently before and after learning; EdgeWrite (Wobbrock, Myers and Kembel, 2003) buys stability and accuracy rather than speed, and its 18 per cent accuracy advantage did not reach significance. Report a speed figure for either of those without saying which strategy participants had reached and you have averaged the design away. Finally, judge the number against something. Kristensson and Vertanen (2014) estimate the inviscid entry rate, where the user rather than the technique is the bottleneck, at 67 words per minute and find most published methods far below it, which is a useful goal and a poor ranking, because the target is a property of people. The comparison that would settle whether your measure transfers is the one nobody in this literature has run: take two techniques, hold the phrase set fixed, and vary the correction policy as a factor, so that the technique by policy interaction can be read off directly. If the ranking survives the change of policy, your measure describes the technique; if it flips, it described the experimenter's instructions.

Begin by naming the two structures in the terms the literature uses, because the argument is about a cost assumption rather than about taste. The breadth prediction is Landauer and Nachbar (1985), whose fitted function of the logarithm of the number of alternatives, a decision component in series with a movement component, favours breadth over depth for a known target. It predicts the deep structure, with its extra decisions but shorter movements, to be slower. The counter-derivation is Lee and Macgregor (1985), whose optimum of four to eight alternatives per page assumes an exhaustive search, against Paap and Roske-Hofstrand (1986), who allow restricted search and obtain 16 to 78 across the same parameter values. Since the broad first panel of twelve sits inside both ranges and the deep first panel of five sits inside one and outside the other, the study has to fix which search model is assumed rather than let the team pick the figure that flatters its design. Run a within-subjects comparison of the two structures on the same 120 commands and the same tasks, and manipulate the search model as a factor: half the tasks name the setting exactly, half describe a goal the user must interpret, following Norman and Chin (1988), who found the effect of tree shape appeared for scenario targets and largely vanished for named ones. Measure more than time. Record navigation steps and error rate, because Kurtenbach and Buxton (1993) found error was the binding constraint for experts and that breadth and depth traded evenly once it was respected; record preference and perceived control, because Findlater and McGrenere (2004) found the static menu faster than the adaptive one while most users preferred the one they controlled; and record search time at first contact and again after repeated use, because Uddin and others (2016) found a large technique slower at the start but equivalent after experience, so a single measurement point would have reversed the recommendation. Hold geometry constant, since Callahan and colleagues (1988) changed only layout and changed both time and error. A breadth result requires the broad structure to be faster or less error-prone for named targets without losing on scenario targets or on early-use time. A depth result requires the reverse. A result for the third position requires the two structures to be indistinguishable once the named and scenario tasks are separated, while item order or geometry produces a difference the structures do not, which would show that the breadth-depth axis was never the operative variable here.

Fix what the participant is told before they act, because that is the whole argument. Every elicitation study in the unit portrays the effect of a command and then asks the participant to perform its cause, which is why Wobbrock, Morris and Wilson (2009) can report agreement at all: the referent is already understood when the gesture is proposed. That design measures reassignment, and the only defensible way to measure discovery is to present the interface without instruction and observe whether the correct gesture is produced on the first attempt. So I would run two conditions on the same command, one where the effect is portrayed and one where it is not, and I would report the first-attempt success rate separately from the agreement rate, because the two answer different questions. I would take the gesture set from an elicitation study rather than from my own preference, since Wobbrock, Aung, Rothrock and Myers (2005) show that guessability is a measurable quantity that can be improved, and I would measure it between people rather than within a person. For the gesture set itself I would need more than one number. Vatavu and Wobbrock (2015) add a disagreement rate for referents and a coagreement rate between referents, and Vatavu and Wobbrock (2019) show that a low rate can come from an unclear referent rather than an unguessable gesture, so I would report the pair and check referent clarity before reading a low rate as a verdict. I would measure retention after a delay as well, because Nacenta, Kamber, Qiang and Kristensson (2013) find user-defined gestures easier to remember, with up to a 24 per cent recall advantage, and that advantage begins only after learning, so it is a different property from first-encounter discovery. The setting matters to what I could conclude. Hinrichs and Carpendale (2011) found gesture use shaped by social and interaction context, with gestures forming sequences, and an elicitation study asks one person for one command at a time, so a laboratory result would describe a laboratory. If the interaction is mid-air I would expect the debt to be larger, since Piumsomboon, Clark, Billinghurst and Cockburn (2013) take the method off the surface, and Vatavu (2012) shows the resulting set is specific to its domain, while Liu and colleagues (2024) show agreement can be raised by refining referents, which means a poor result may indict my referents rather than the gesture. Finally I would be clear about what the result licenses. Shneiderman (1983) and Hutchins, Hollan and Norman (1985) locate the advantage of direct manipulation in a visible object and a short distance between the user's goal and the interface's expression, and Norman (2010) shows that a gesture supplies neither until familiarity makes it behave like a label. So a high first-attempt rate would license shipping the gesture alone for that command in that setting; anything less would licence shipping it as an accelerator beside a visible control. To test transfer I would repeat the no-instruction condition on a second command and in a second setting, because the unit's own evidence says agreement is command-dependent and context-dependent.

I would begin by fixing the role, because the number of patterns a wristband can carry means nothing until the role is named. Jones and Sarter (2008) and MacLean (2008) show that touch suits alerting, confirmation, attention management and guidance rather than the expression of arbitrary state, so the set should encode turn, hazard and message and not a vocabulary of interface modes. From Brewster and Brown (2004) I would take the structure of a tacton, building each pattern from frequency, amplitude, waveform, duration and body location, and I would state the number of dimensions and levels rather than proposing a set of unanalysed buzzes. Following Yatani and Truong (2009), whose participants distinguished ten patterns at approximately 90 per cent accuracy without looking, I would set a candidate vocabulary near ten and treat that figure as the benchmark to beat under load, not as a ceiling. The set would be defined twice: fixed for everyone, and generated for each participant in the manner of Hazra and Shiakolas (2024), who found personalised feedback preferred and success reaching 100 per cent after only three repetitions, with reported mental demand falling. Because Burke and colleagues (2006) show that visual-tactile feedback helps most when several tasks run at once and workload is high, I would test identification while the participant performs a competing visual task, which is the condition the cyclist is actually in, and I would measure first-attempt identification rate, reaction time, confusion between specific pairs and reported workload, rather than accuracy alone. I would also record actuator specification, since Hoggan, Brewster and Johnston (2008) found that better hardware improved performance further, and that difference would otherwise be attributed to the patterns. Bergamasco (1995) supplies the boundary of the claim: a channel that must let a hand feel an object's resistance is a manipulation demand, and my result would say nothing about force feedback. The result would license a statement about how many patterns this set, on this device, can carry under divided attention in this population; it would not license a general bandwidth figure, because the set, the hardware and the load are all part of the measurement. To separate a channel limit from a pattern-set limit I would run the same task with the set fixed and with it personalised, within the same participants, under the same load.

The team's reasoning fails at its first premise. Nielsen (1993) states the three limits as limits on the user's state: 0.1 second for the system to feel as though it is reacting instantaneously, 1.0 second for the flow of thought to stay uninterrupted, and 10 seconds for attention to remain on the dialogue. Those describe when a person notices and when they begin to disengage, not how much delay an operation may carry, and Miller (1968) refused exactly this move when he argued that the controversy cannot be resolved without asking response time to what, because different purposes and actions have different acceptable or useful response times. A budget of 50 milliseconds is therefore an answer to a question the team has not asked. The measurement literature shows what the numbers do and do not cover. Ng and colleagues (2012) found users greatly preferred lower latencies in a touch demonstrator, with noticeable improvement continuing well below 10 milliseconds against a commercial baseline of 50 to 200 milliseconds, so detection is far below the folk limits, while Yang and colleagues (2025) place the perceptual threshold in virtual reality at 130 to 170 milliseconds, moving with the strategy and the difficulty of the task. Jota and colleagues (2013) found performance in direct-touch dragging decreasing as latency increased, more severely for smaller and more distant targets, which means the same millisecond figure costs different amounts in different parts of a canvas. Deber and colleagues (2015) showed detection depends on form factor and task, and Long and Gutwin (2019) found different input paths behaving differently under the same latency. A single figure cannot describe that spread. Martens and colleagues (2018) matter most for the sign-off decision. In a second-order control task, latency of 14 milliseconds already affected performance, and performance and perceived control difficulty were affected before participants were aware of the latency. A test that asks users whether the drawing feels delayed measures the most forgiving threshold available. MacKenzie and Ware (1993) show why a fixed budget also fails arithmetically: at 225 milliseconds of lag, movement times rose 64 per cent and error rates 214 per cent, and a model in which lag multiplies the index of difficulty explained 94 per cent of the variance, which Hoffmann (1992) had predicted from a modified Fitts's law. Delay multiplies the cost of a difficult target rather than adding a constant, so a budget that is generous for a large brush stroke is tight for a small precise one. Pavlovych and Stuerzlinger (2009) add that latency and jitter have different effects, the second raising errors roughly in inverse proportion to target size while the filtering that removes it adds lag, and Warburton and colleagues (2022) show a latency figure is a property of device, movement phase and method, with motion prediction reducing a measured 21 to 42 milliseconds to a functional 2 to 13 milliseconds. Beigbeder and colleagues (2004) set the outer bound on tolerance: in a networked game, typical latency, while sometimes unpleasant, did not significantly affect the outcome, so tolerance belongs to the task. The study for this product should therefore be run inside the product's own task family. I would take three operations with different error tolerance, a precise tracing operation, ink stroke entry and canvas panning, and cross them with four latency levels: near zero, below the detection threshold, above it but within the commercial range, and clearly perceptible. Each participant would complete both a detection measure, comparing pairs of delays, and a performance measure, completion time and error, and I would manipulate the baseline experience between groups so that one group works with a near-zero-latency build before the test and another group arrives from a consistently slow build, which turns adaptation from an inference into a comparison. Reporting would name the device, the movement phase, the jitter as well as the mean, and the target sizes used. The result would license a latency profile for these three operations on this tablet with these targets and this baseline, and it would not license a single budget for the product, a general claim about perception, or any statement about other input paths such as a stylus with a different sampling rate. The sign-off artefact should be that profile, with the conditions printed beside it.

The team is right about the population and wrong about the scope of the claim. Card, Moran and Newell (1980) predict the execution time of an expert performing a routine task, and trained data-entry staff doing repeated work are close to that case: the model's operators, keystroking, pointing, homing, mental preparation and system response, were tested at a root-mean-square error of 21 per cent across 32 tasks on 10 systems. The restriction the same paper states is the problem for the team's reasoning: the method must be fully specified in detail and the performance must be error-free, and the model predicts execution time only, not task acquisition. A clerical application whose users must read work items from a queue or interpret ambiguous entries has acquisition costs the model does not cover. The strongest evidence for the model is also evidence about what a validation establishes. Project Ernestine (Gray, John and Atwood, 1993) collected four months of data over 72,450 calls and showed GOMS predicting, against expectation, that a proposed workstation would be slower; the difference varied with call category, and the predicted slowdown was worth almost two million dollars a year. That is a comparison between two designs on an expert routine task, which is exactly the decision the team faces. It is not a forecast of absolute times. Gong and Kieras (1994) separate the two: a 40 per cent predicted improvement in execution time against 39 per cent observed, and a 46 per cent predicted learning reduction confirmed at 46 per cent, but absolute execution times overpredicted by a factor of more than two until the estimation rules were corrected, after which the error fell within 9 per cent. A single uncorrected analysis would therefore mislead the team about how long data entry will take. John (1995) shows how much the method can carry, and where it stops. The analysis itself took 12 per cent of project time and about a fifth of the interface coding time, and she reports a map-digitising case predicting a 73 per cent task-time reduction worth 730,000 dollars, and states that for cognitive-skill aspects the validity of GOMS is no longer in question. She also warns that error prediction is not mature enough for designers without formal psychological training to use. John and Kieras (1996) set the boundary: GOMS applies only to routine cognitive skill, the analyst must bring the user's goals from outside the model and cannot correct a wrong list, and no method for predicting what errors users will make has been validated as an engineering tool. A clerical system where mistakes are costly and the redesign might change which entries are misread is precisely where the model is silent. Modelling can also explain, not only predict. Hornof and Kieras (1997) used a cognitive model to show that menu search is both random and systematic, and Byrne (2001) reached a zero-parameter fit, excellent though not perfect, on menu selection with the ACT-R/PM architecture. Neither licenses a claim about data-entry accuracy. The evaluation I would run keeps the analytic and the empirical in their proper roles. Let the GOMS analysis decide the comparison between the two candidate layouts, where it is strongest, and require it to report the operator assumptions and the calibration it relies on. Run a usability and error study with representative staff on the chosen layout to cover acquisition, errors and recovery, which the model does not predict. Then settle the disputed question: have an independent analyst model the same interface blind to the empirical result, and compare the predictions against the study, so the team learns whether the model ranked the designs for the right reason.

The team's request contains a reasonable premise and an unreasonable one. The reasonable premise is that heuristic evaluation is worth running: Molich and Nielsen (1990) surveyed seventy-seven designers and programmers and found that identifying specific, potential problems in a dialogue design is difficult, and Nielsen and Molich (1990) showed that review returns real problems cheaply. The unreasonable premise is that one evaluator's ranked list can carry the authority of a number. In the founding study an individual evaluator found between 20 and 51 per cent of the problems present, and it was aggregating three to five evaluators that did rather well, so the unit of inspection is a panel rather than one expert. With budget for one evaluator I would either buy three to five cheap evaluators or relabel the output. The reliability evidence fixes the limit. Hertzum and Jacobsen (2003) reviewed eleven studies and found that two evaluators using the same method on the same system agreed on 5 to 65 per cent of the problems they reported, with the effect holding for novice and experienced evaluators, for cosmetic and severe problems, and for detection and severity alike. Jacobsen, Hertzum and John (1998) had four evaluators analyse four videotaped test sessions and found that only 20 per cent of ninety-three problems were detected by all four and 46 per cent by a single evaluator, so the variance is not confined to inspection. That is why I would report agreement rather than a ranking. Severity is the part the team most wants and the part least able to bear weight. Nielsen (1992) found single-evaluator severity ratings very unreliable, but the mean of four evaluators came within half a rating point of the true severity 95 per cent of the time. So severity is usable only as an aggregate, and only from several evaluators. The list is a set of hypotheses about the interface. John and Mashyna (1997) had an analyst learn and apply the cognitive walkthrough to a multimedia authoring tool and found it learnable and flexible but disappointingly ineffective at predicting the actual user problems. Jeffries, Miller, Wharton and Uyeda (1991) found that four techniques overlapped only partly, and Desurvire, Kondziela and Atwood (1992) report that laboratory testing identified four to five times as many problems as heuristic evaluation. Gray and Salzman (1998) examined five influential comparisons and found design faults that call into question what the field thought it knew, so I would not claim that inspection outperforms testing or the reverse. To the team I would deliver a candidate list, a statement of how many evaluators produced it and how far they agreed, and a separate short list of the hypotheses that a test with real users should confirm. I would present severity as an interval from an aggregate rather than as a rank, I would mark each item as observed or inferred, and I would refuse to claim that the list is complete, that the ranking is a measurement, or that fixing every item will make the site usable.

The times can be reported as a description of the sessions that produced them, not as a property of the two designs. Hertzum, Hansen and Andersen (2009) found that relaxed thinking aloud lengthened tasks, moved more time into distributed visual behaviour, increased navigation commands and raised reported mental workload, while classic thinking aloud had little effect apart from prolonging tasks. Boren and Ramey (2000) show that the relaxed form is what practice uses even though it is not what the theory invoked to license think-aloud describes. The timings are therefore the durations of doing the tasks while explaining them. The comparison with the competitor's data is not available at all. That study used a different protocol, no moderation and a different setting, so the difference between the numbers confounds the design with all three. Duh, Tan and Chen (2006) found laboratory and real-life tests of the same device differing significantly in the frequency and severity of problems, in user behaviour and in subjective responses, and Kjeldskov and Graham (2003) show how heavily the field's evaluations default to the laboratory, which makes the setting a choice to be stated rather than a neutral frame. Within the team's own study the comparison between the two prototypes is the defensible claim, and even there the report should name the regime and the setting that produced it. The problem lists are the more portable part of the evidence: Van den Haak and colleagues (2004) found the approaches very much comparable on the problems surfaced, Krahmer and Ummelen (2004) found the number of navigation problems unchanged by protocol, and Alhadreti and Mayhew (2018) found concurrent think-aloud detecting more problems than retrospective at much lower evaluator cost. A list is still only as good as the procedure that produced it: Hornbaek and Frokjaer (2008) found about 40 per cent agreement between evaluators and identified the matching of problems as a key determinant of that effect, so the report should state the coding procedure and the number of analysts, in the manner Doi (2021) proposes, and say how far they agreed. I would refuse three claims: that the times measure the prototypes rather than the sessions; that the two studies can be compared; and that the problem list is complete. I would also put Greenberg and Buxton's (2008) question to the team about whether this design was the right instrument, given the analysis cost that Kjeldskov, Skov and Stage (2004) show can be reduced without weakening the test.

The score can support a claim about perceived usability and nothing more. Brooke (1996) began from the position that usability is not a quality that exists in any real or absolute sense and cannot be specified without naming the users and the tasks, and Bangor, Kortum and Miller (2008) describe the scale as collecting a user's subjective rating of a product's usability. The number the team publishes is therefore a judgement of the app, not a property of it. Lewis and Sauro (2009) add that the single value conceals structure: factor analysis of two independent data sets produced a usable factor and a learnability factor, so the app may score well on one and poorly on the other while the single number hides the difference. The competitor comparison needs a matching group. The usual SUS benchmark is a mean of 68 with a standard deviation of 12.5, and Hyzy and colleagues (2022) tested whether it transfers to digital health apps and found reason to question the transfer, because the scale was not designed for mobile apps or for that domain. The honest comparison is against similar apps, and the report should name the group. The score also omits what the field calls usability. Hornbaek and Law (2007) meta-analysed measures from 73 studies and found effectiveness and efficiency correlated at .247, efficiency and satisfaction at .196, and effectiveness and satisfaction at .164, so the aspects are largely independent. If completion rate or time on task matter, they belong in the report beside the score rather than behind it. The five-user plan rests on a model, not a rule. Nielsen and Landauer (1993) fitted problem detection across eleven studies as a Poisson process, and Virzi (1992) reported that four or five participants detected around 80 per cent of the problems. Lewis (1994) showed that the eighty per cent figure holds only while the average likelihood of detecting a problem lies between .32 and .42, and Faulkner (2003) sampled sets of five from sixty users and found that some sets found 99 per cent of the problems while others found only 55 per cent. Schmettow (2012) calls such magic numbers hocus-pocus, and Borsci and colleagues (2013) propose a grounded procedure for choosing the sample instead. The team should state the coverage it needs, estimate the detection rate from early sessions, and size the study to that estimate, reporting the procedure and its uncertainty beside the results.

A strong answer separates the levels and says what each one buys. The rule itself, which is a decision table, gets unit tests: many of them, cheap to keep, covering each branch including the case where the rule data is missing. The claim endpoint gets an integration test against a test database, because the wiring between the rule and the stored record is what a unit test cannot see. A small number of end-to-end journeys cover the three outcomes a claimant can meet: eligible, ineligible, and unable to complete. Continuous integration runs all of these on every merge, and the rule's configuration is reviewed and tested with the code so that the release candidate is tested in the state it will ship in. Release is separated from deployment: the rule sits behind a flag, the deploy ships with the flag off, and the flag is enabled for a small share of traffic in one region first. The signals should be named in advance and should include the error rate of the endpoint, the completion rate of the claim journey, and the distribution of decisions compared with the old rule for a sample assessed under both. The stop rule must be written before the flag goes on, and one named person must be able to reverse it. Two judgements earn the marks: who reads the signals, and when the flag will be removed, since a flag with no removal date is a configuration problem waiting to happen.

The record should begin with the decision in one sentence: the server is the authority on whether a slot is held, and the client holds a token plus a copy for display only. The boundaries follow. Nothing in the interface may decide availability, because that is a claim about truth; anything that changes a hold crosses the network boundary and is recorded there; the screen that displays the hold can be replaced without changing the rule. The rejected alternative is a browser-only hold, which fails because the client becomes the authority and two people can be told they hold the same slot; the record should also record the alternative of server session state, which loses the hold when a connection drops and, in Fielding's terms, gives up the visibility and recovery properties that stateless communication buys. The non-functional requirements belong in the same record, because they cannot be retrofitted: a stated time within which a hold must be acknowledged, a stated conflict error, a log that makes a failed booking reconstructable, and the accessibility requirement that the hold countdown and the conflict message are announced to a screen reader, added to the component's acceptance criteria rather than to a later audit. The review comment should attack the claim rather than the person, ask for the measurement of speed on a slow connection, require that the double-hold case is shown to be impossible, and require that the record is updated if the change is accepted.

Begin by separating the two claims. Unchangeability is a claim about design history, and Parnas (1994) gives it a testable form: programs age because successive changes accumulate in structures that were not designed for them, so the evidence is the effort each change now costs relative to the effort it cost when the structure was new, together with the number of interfaces a change touches. Poor measurement is a claim about classification, and Swanson (1976) supplies the remedy: report corrective, adaptive and perfective work separately, because a single maintenance number cannot show whether the problem is defects, a moving platform or the team's own deferred improvements. A large unsegmented figure can be manufactured by classification rather than discovered. The reason the literature cannot adjudicate the growth of cost is that the field's regularity, Lehman's laws, is contested. Herraiz and others (2013) found controversy about the laws common, left open after about fifteen years without revision, and drawn from a narrow base of systems. The study that would settle it is the design Kemerer and Slaughter (1999) show at product scale, applied across organisations: more than 25,000 change events across 23 systems over 20 years was one study, and the missing half is many organisations using categories declared in advance and applied identically, followed long enough for phases to appear and reported so that a failed prediction shows up as a failure. Brooks (1987) is the caution on the rewrite: the essential difficulty of changeability is not removed by any single development, so a new system does not escape the time horizon, it resets the clock.

Begin with fidelity, because the unit's first claim is that a method change is a change in who decides what and when. The Scrum Guide defines the team as self-managing, deciding internally who does what, when, and how, with one accountable Product Owner ordering the backlog (Schwaber and Sutherland, 2020), so fidelity should be measured against those decision rights and the review cadence, not against ceremony attendance. Expect the enacted process to differ from the documented one, since Hoda, Noble and Marshall (2013) found six informal roles, including Mentor, Coordinator and Champion, doing work the methods assume but never specify; record who actually performs those functions. For the organisation, compute coordination requirements from the technical dependencies between the files each team touches and compare them with observed coordination, because Cataldo and others (2006) show that congruence between the two shortened development time. Do not trust self-rated project success as the primary outcome: Serrador and Pinto's (2015) 1002-project sample was recruited through professional networks and LinkedIn groups and asked respondents to describe a more and a less successful project they had already worked on. Prefer measures drawn from artefacts, such as post-release failures, where organisational metrics predicted failure-proneness better than code metrics in the Windows Vista study (Nagappan, Murphy and Basili, 2008), and assert nothing about the amount of planning the teams do without measuring it, since the same survey found no difference in upfront planning and planning that continues throughout. Avoid the two-way contrast altogether: with 52 per cent of projects hybrid (Gemino, Horner Reich and Serrador, 2021), compare blends and record which decisions are devolved. Finally, state the limit. With three teams and no instrument for adoption you cannot exclude selection, so the honest conclusion is an association, which is why the field still needs the common agenda Dyba and Dingsoyr (2008) set out and the theory-based approach Dingsoyr and others (2012) demanded.

The claim under test is the unit's: a criterion forces a decision, so it should change what is built and what is caught, not merely what teams say they prefer. Define the criterion before the rule starts, because the interesting content is the part teams omit. It must state a condition a test can check, and the definition must cover non-functional content and failure paths, since Ramesh, Cao and Baskerville (2010) found non-functional requirements such as security and scalability among the risks that agile requirements practice left intractable, and since a story requires no failure path at all by comparison with a use case (Cockburn, 2001). Measure from artefacts rather than recollection: post-release defects, rework on stories already accepted, and the exceptions that reach production unspecified. Record which criteria exist and who wrote them, so the criterion is the variable and the format is held constant; that is the design Ferrari and others (2018) come closest to, and only in one company. Avoid the weakness of the perception literature, because practitioners who chose stories and INVEST report them more favourably (Lucassen and others, 2016), and the format always travels with Scrum. Introduce the rule in steps across teams and keep the log for all of them. Treat Damian and Chisan (2006) as a mechanism to look for rather than an effect size to expect, since it is one 30-month case. Finally state the limit: with one product and no instrument for adoption you cannot exclude selection, so the honest conclusion is an association, and the criterion's authority rests on the decisions it forces.

The record should begin by replacing the name with a decision. The claim being made is that a given set of parts may change and be deployed independently of the rest, which can be enforced by a pipeline: a service that cannot be built and released on its own fails the constraint, and a release train that always moves every service together falsifies it (Fowler and Lewis, 2014). Name the parts and name the enforcement, because a boundary nobody can enforce is a drawing. Second, state the price in units the team can observe. Decomposition moves data management into a distributed setting, so the record should name the measured quantity: cross-service transaction complexity and query complexity as recorded in practice reviews (Soldani and others, 2018), latency and consistency trade-offs at the data layer (Laigner and others, 2021), and the operational cost of running more services and their runtime environments. Read those numbers from artefacts rather than recollection: pipeline logs, incident records, latencies and the count of cross-service transactions per feature. Third, state what would change the team's mind: if the measured operational cost per released feature rises while the number of independent releases stays flat, the boundary is charging more than it buys. Fourth, hold the pattern's role fixed. Class-level evidence shows that stability depends on the role a class plays and on how many pattern occurrences it is coupled into, and that pattern classes do not differ significantly from non-pattern classes overall (Ampatzoglou and others, 2015), while the service vocabulary itself varies by migration, orchestration, storage and deployment setting (Taibi and others, 2018). A pattern name imported from another setting therefore carries no price of its own. What the record cannot establish is that the split caused any later change in quality, because the team is one case, the practice literature behind the price list is grey literature reporting recognition rather than measurement (Soldani and others, 2018), and no study in this unit holds implementation, role and outcome measure constant at once (Wedyan and Abufakher, 2020).

The record should begin by naming the trade, because a policy that is described as a preference cannot be reviewed. Keeping branches for a fortnight buys isolation from interruption: after branching became cheap in distributed version control, branch creation rose from 1.54 to 3.67 per project per month, and observed branches in the Linux kernel were significantly more cohesive than simulated branches over the linearised history, with reduced interference between strongly coupled tasks (Chacon and Straub, 2014; Barr and others, 2012). What the policy carries is deferred integration: developers in a large industrial project named delay in moving changes between teams as their central problem, attributed to having too many branches, and removing high-cost, low-benefit branches there would have saved 8.9 days of delay per change at the cost of 0.04 additional conflicts on average (Bird and Zimmermann, 2012). Second, name the quantities and where they are read. From the repository: branch lifetime, divergence from trunk and the number of merges per release. From the pipeline: the size of the merged change and whether the checks ran on the merged state or on the branch alone. From release and incident records: post-release failures and rework on the components touched on integration day. Conflict counts are a weak indicator on their own, since conflicts occurred in 9.38 per cent of merges and mostly came from edits to the same or consecutive lines of one method (Accioly and others, 2018). Third, state the falsifier: if integration-day merges grow larger while reviews stay flat and post-release failures rise on the touched components, the policy is charging more than it buys. Fourth, record the competing explanation, since the factors that decide a merge are largely not technical and integrators struggle to prioritise contributions (Gousios and others, 2014; Gousios and others, 2015); a lengthening queue may be an attention problem rather than a branching one. What the record cannot establish is causation, because no study here holds branch lifetime, implementation and review attention constant together, and the closest quantitative estimate is a simulated counterfactual for one product (Bird and Zimmermann, 2012).

Start with what the number measures. Inozemtseva and Holmes (2014) generated 31,000 test suites for five Java systems of up to 724,000 lines and found that, with the number of test cases controlled for, coverage correlated only low to moderately with effectiveness, and that stronger forms of coverage gave no greater insight than weaker ones. An 85 per cent target therefore buys a reliable warning about under-tested code and no assurance that the suite will catch a change. Second, name the property the suite should hold and where to read it. The unit's answer is a deterministic instrument and a portfolio of techniques: Basili and Selby (1987) found that code reading, functional testing and structural testing detected different classes of fault, so the mix matters, and Gruber and others (2021) found 7,571 flaky tests among 876,186 Python test cases, with order dependency causing 59 per cent and test infrastructure 28 per cent, so the artefacts to read are the failing tests, their rerun history and their repair rate, not a percentage. Parry and others (2021) report that 59 per cent of developers deal with flaky tests monthly, weekly or daily, and frame flakiness as a threat to any method that assumes an outcome depends only on the code it covers. Third, say what test-first is and is not evidenced to deliver. Rafique and Misic (2013) pooled 27 studies and found a small positive quality effect with no discernible productivity effect; George and Williams (2003) found 18 per cent more functional black box test cases passed at the cost of 16 per cent more time. Adopt the rule for the feedback and the design pressure Beck (2002) describes, not for a promised speed or quality win. Finally, state the falsifier: if merge throughput falls and the defect count does not, the rule is not paying. The note cannot establish that test-first caused any outcome; Fucci and others (2017) found sequencing had no important influence while granularity and uniformity carried the effects, so the honest version buys the discipline and measures the result.

Start with the definition, because the proposal rests on it. Humble and Farley (2010) separate deployment, which installs a version of the software into an environment, from release, which exposes that version or a feature inside it to users, and they introduce the deployment pipeline as the automated path from check-in to a release-ready build. Fowler (2006) draws the consequence: with a release-ready mainline the decision to release the latest version into production becomes purely a business decision, a matter of minutes running an automated pipeline. That is the gain the team is buying, so the note should say plainly that it moves the decision from engineering to the product owner. Second, be honest about the evidence. Hilton and others (2016) analysed 34,544 GitHub projects, 1,529,291 builds and 442 developers and found evidence that continuous integration helps projects release more often and that adoption keeps growing. Vasilescu and others (2015) associate continuous integration with higher productivity and no observable quality loss, but the design is observational and cannot separate the practice from the teams that choose it. Rodriguez and others (2017) mapped 50 primary studies and concluded the topic is still in its infancy, and Shahin and others (2017) found the reviewed literature dominated by validation and evaluation designs. The capability is demonstrated; the comparison is missing. Third, name the cost and where to read it. Rahman and others (2016) analysed 39 Google Chrome releases across five years and found that toggles reconcile rapid releases with long-term development but introduce technical debt and additional maintenance, and Fowler (2006) instructs teams to remove flag logic promptly so that flags do not clutter the code base. The artefacts to read are the flag inventory, the age of each flag and the removal rate, not the deployment count. Zampetti and others (2020) add that the pipeline itself carries faults, cataloguing 79 bad smells in 7 categories from 13 expert interviews and more than 2,300 Stack Overflow posts. Fourth, state the falsifier: if release frequency rises, but the flag inventory grows faster, or pipeline faults show up as incidents, the separation is relocating risk rather than removing it. The note cannot establish that continuous integration caused any of the reported outcomes, and it cannot confirm that the flags will ever be pruned, because no cited study reports a removal rate. Fifth, say who the release may not reach: a user whose older handset, metered connection or assistive client cannot follow the release sits outside the separation, so the artefact to read is the spread of client versions still in use.

Start with what the yield actually is, because the manager's premise is half right. Bacchelli and Bird (2013) observed 17 developers across 16 Microsoft product teams, classified 570 comments in 200 threads and surveyed 165 managers and 873 programmers; defect comments were only the fourth most frequent of nine categories, at 78 comments or 14 per cent, while code improvements led with 165 comments, 29 per cent. That is a count of comments, not of escaped defects, and the same study says defect comments mainly cover small logical issues while knowledge transfer, team awareness and alternative solutions appear as additional outcomes, with understanding the key aspect of reviewing and only 12 comments explicitly about knowledge transfer. Second, concede how weak the counter-evidence is. Mantyla and Lassenius (2009) classified 388 and 371 defects from nine industrial C and C++ reviews and 23 student Java reviews and found that three quarters did not affect visible functionality at all; their largest group, evolvability defects, made code harder to understand and modify, which is precisely what no test detects. So the proposal would remove the only mechanism in the toolchain aimed at that class of problem. Third, put the outcome evidence in front of them with its limit named. McIntosh and others (2016) studied Qt, VTK and ITK and found review coverage, participation and reviewer expertise each share a significant link with post-release defects, and the design is observational, so it cannot separate the practice from the teams that sustain it; more importantly, it measures the process, so cutting coverage moves exactly the variable the study links to quality. Fourth, name the social mechanism rather than treating review as a queue. Thongtanunam and others (2015) found reviews with a code-reviewer assignment problem take 12 days longer to approve a change, and propose RevFinder, which recommends reviewers from similar reviewed file paths, so most of the reported bottleneck may be assignment rather than review. Bosu and Carver (2014) add that reputation decides as much as content: across eight open-source projects, core developers got earlier feedback and more acceptances while peripheral developers waited 2 to 19 times longer, between 12 and 96 hours, and Baysal and others (2016) found in WebKit and Google Blink that patch size, priority, the organisation behind the contributor and the reviewer's load and experience all significantly affect response time and outcome. Fifth, offer a falsifier and an honest limit. I would change the recommendation if comment-level data showed that code-improvement comments, not defect comments, predict the same post-release defects that a stronger test suite already catches, or if mandatory review could be turned off in one organisation and post-release defects did not rise while a validated measure of understanding held steady. What no instrument in this unit measures is the shared understanding itself: Rigby and Bird (2013) introduce a knowledge-sharing measure and they are the exception, and the automation work of Tufano and others (2021) targets the local comment, which is the part of review that is least in dispute.

Start with what the metaphor actually claims, because the pilot has been treated as free. Cunningham (1992) wrote that shipping first-time code is like going into debt, that a little debt speeds development if it is repaid promptly with a rewrite, and that an unconsolidated implementation can bring an organisation to a standstill, so the hard-coded rule is a loan rather than a shortcut. Fowler (2009) supplies the test: a prudent debt taken to reach a release may not be worth paying down if its interest is small, but interest paid on every later change is the opposite case. Second, say how rarely practice matches that discipline. Ernst and others (2015) surveyed 1,831 engineers and architects at three large organisations and found architectural decisions the most important source of debt while doubting that practice carries the metaphor beyond communication; Albuquerque and others (2022) surveyed 120 practitioners from 86 organisations and found debt widely recognised, with 72 per cent able to give a valid example, yet formal measurement and documented repayment plans rare. Third, name what repayment does and does not show. Kim and others (2014) found the top 5 per cent of preferentially refactored Windows modules reduced inter-module dependencies by a factor of 0.85 while the rest increased theirs by 1.10, but the same study shows impact needs several dimensions and cannot attribute defect reduction to refactoring alone; Silva and others (2016) add that refactoring is driven mainly by changing requirements rather than by code smells. Fourth, the interest compounds at the architectural scale, and Li and others (2015) offer a method that identifies such debt from the gap between decisions and later change scenarios before the code exists. Fifth, the newest debt is hidden: Sculley and others (2015) show machine learning systems accrue maintenance costs through boundary erosion, entanglement and undeclared consumers that no linter inspects. The falsifier is longitudinal evidence that a recorded, owned repayment plan reduces later change effort; Lenarduzzi and others (2021) show that such rigorous evaluation is still missing, so schedule the repayment with a date and an owner rather than leaving the ticket open.

Begin by treating each attribute as a decision taken before construction, not an inspection after it. Chung and others (2000) model quality attributes as softgoals refined and traded off against one another, so the first step is to state the tradeoff explicitly, for example that a richer checkout animation will not be bought at the cost of the time to first interaction. State the accessibility criterion in the form the standard gives. WCAG 2.2 is a set of testable success criteria that are technology-independent, and it says itself that it will not address every user need and is testable only with a combination of automated testing and human evaluation (World Wide Web Consortium, 2024), so write criteria such as a minimum contrast ratio and a keyboard-reachable focus order as automated checks, and add a screen-reader journey through the whole checkout as a human check. Justify the human half from the numbers: Power and others (2012) found that only 50.4 per cent of the problems blind users met on 16 websites were covered by WCAG 2.0 success criteria, and Vigo and others (2013) found that automated tools cover at most half the criteria and catch only about four in ten of those, so a tool-only criterion is of unknown strength. For performance, follow Woodside and others (2007), who argue performance must be predicted from models early and measured later, so set a budget for the checkout's first response on a target device and predict it before building, then fail the build when the budget is exceeded; Jin and others (2012) show performance defects recur as recognisable patterns, so the budget can target known failure modes. For observability, follow Beyer and others (2016), who define a service level objective as a target value for a measured service level, and write an objective such as a maximum rate of failed checkouts and a target time to detect a failure; Yuan and others (2012) show logging is pervasive but arbitrary and often wrong on the first attempt, so instrument the checkout as part of the work rather than adding logs when it breaks. A falsifier: if a longitudinal record showed that meeting these criteria did not track the user-facing outcome, for example task success for disabled users or the time to diagnose a real failure, then the criterion would need replacing rather than the outcome relaxing.

A good answer commits to one theory and then shows what the commitment costs. Choosing distributed cognition means treating the team, the chart, the two systems and the pager as one cognitive system, so the analysis asks how information is transformed between representations and where a transformation loses the thing that matters (Hutchins, 1995). The alternative to argue against should be named and given its due. GOMS is the wrong instrument here, because handover is interruptible, involves several people at once and is not one practised procedure; its simple variants assume serial processing with no interruption and no interleaving of goals, and its validated predictions sit at the keystroke level (John and Kieras, 1996). Activity theory is a strong rival, and the answer should say why it was not chosen: it directs attention to the object of the activity and to contradictions between groups, which explains well why the two software systems hold different pictures of the same patient. Distributed cognition was chosen because the errors cluster at the transformation of representations, which is what that theory describes. What the analysis will not establish is a general claim. One ward studied in detail supplies mechanisms and hypotheses, not an effect size for another hospital, and the answer should say what would change the choice.

The first move is to separate execution time from learning. A keystroke-level model predicts execution time for the method the analyst supplies, and cannot show that anybody performs that method, so it does not support a claim about ease of learning (John and Kieras, 1996). The variants that predict learning require an explicit account of procedural knowledge, and their validated predictions, in the paper that compares them, sit at the keystroke level of analysis. The second move is to separate a model output from a measurement. The 15 per cent is the model's number, so the evidence for it is the model, the operator times and the assumed method. I would ask for the model itself, the source of each operator time, and the exact method assumed for the task. Then I would ask whether a first-time user of a public service plausibly follows that method, since a public booking app is used occasionally and often by people who are not practised. I would ask how the two designs were compared, because a comparison of model outputs rests entirely on the procedures assumed, and I would ask what the 15 per cent is 15 per cent of, since a few seconds saved in a yearly task is a different claim from a few seconds saved in an hourly one.

A keystroke-level analysis predicts the execution time of a method the analyst supplies, for an expert, without error (Card, Moran and Newell, 1980); it says nothing about acquisition, choice between methods, or error recovery, and it excludes error-containing tasks even though up to a quarter of an expert's time can go on corrections. The team's claim is therefore too strong in one direction and too weak in another. What the tradition can establish is genuine: Project Ernestine shows GOMS models predicting, before a system was built, that a proposed telephone-operator workstation would be slower than the incumbent, and a four-month field trial of 72,450 calls confirmed it at a cost of almost $2 million a year (Gray, John and Atwood, 1993). But the claim that testing is unnecessary assumes the modelled procedure is the one users will follow. Bhavnani and John (1996) found that experienced computer-aided drafting users relied on suboptimal strategies resembling manual drafting, because the strategic knowledge for using the tool well was neither defined nor taught; a keystroke-level prediction can be exactly right about a procedure no efficient user follows. Evidence that routine performance is assembled from microstrategies chosen in the moment (Gray and Boehm-Davis, 2000) reinforces the point, and Olson and Olson (1990) record that the models do not extend to context, fatigue, individual differences or mental workload, all of which matter in clinical work. A study to test the claim should take the intended users rather than expert proxies, compare the predicted method with the methods they actually adopt, observe the interface in use over weeks rather than a single session, and measure execution time, errors and method choice together. A model grounded in the observed methods, whose prediction matches the field data, would count for the team; a match for the analyst's ideal method but not for the methods users adopt would count against it.

An architecture is a claim about what is invariant in one agent, so it can supply a great deal of the proposed analysis. The mature ACT-R theory integrates perceptual-motor, goal and declarative memory modules that place chunks in buffers for a production system selecting a single rule at any moment, and its examples across choice, latency and brain data need little parameter estimation (Anderson and colleagues, 2004). Rational analysis would let the trust predict from the structure of the environment rather than from a model of internal mechanisms (Anderson, 1990), and Anderson and Schooler (1991) show that the need for an item tracks frequency and recency, which is exactly the statistics of a ward's repeated drug names. EPIC shows that the programme can reach concurrent activity, since its multiple-task models rest on adaptive executive control (Kieras and Meyer, 1997). What the architecture cannot supply is the handover itself: how the work is divided between two nurses, the chart and the screen. Byrne's final menu selection model reached its zero-parameter fit only with guidance from eye-tracking data on people doing the task (Byrne, 2001), and Howes and Young's model takes its difficulty from the consistency of the device's mapping while the authors state that it is no powerful learner and does not model error recovery (Howes and Young, 1996). Against the claim sit the multiplication and the knowledge-level findings: 84 surveyed architectures with 49 active (Kotseruba and Tsotsos, 2020), an under-specified knowledge level (Lieto, Lebiere and Oltramari, 2018) and an unfinished modelling bridge across the bands (Anderson, 2002). A study should observe the trust's own two-nurse handover for several weeks, with the model's predictions recorded before the observation and without refitting, and measure handover duration, interruptions, dose discrepancies and which person or artefact holds each piece of information. A prediction that matches the observed times and error points counts for the trust; a match for the modeller's ideal procedure but not for the procedures the nurses adopt counts against it.

The system-level answer starts from Hutchins (1995), whose cockpit case shows that the cognitive properties of a distributed system can differ radically from those of its members, with remembering achieved across the crew, the booklet and the speed bugs. The handover's competence is therefore not the sum of two nurses' recall: it is carried by the chart, the checklist and, on the unit's argument, the model's proposed review, because removing a representation would change what the system remembers. Zhang and Norman (1994) support the design claim that difficulty moves with the representation, and Scaife and Rogers (1996) name the mechanism through computational offloading, re-representation and graphical constraining, so the trust should expect the arrangement of chart, checklist and proposal to change what the handover demands rather than merely display it. On the contested question, the unit keeps two literatures apart. Clark and Chalmers (1998) argue that a reliably coupled resource can be part of a cognitive process, while Adams and Aizawa (2001) defend contingent intracranialism and hold that no case has been made for transcranial cognition; Hutchins's claim takes the system as its unit and is not refuted by that objection. The study should observe real handovers at both sites for several weeks, record the model's proposals before each handover and without refitting, and measure handover duration, dose discrepancies, interruptions, and which person or representation holds each item of information. Treating the model as part of the system predicts that removing its proposal degrades the handover in a way the nurses' unaided recall does not repair; if instead the handover is unchanged when the proposal is withheld, the model is a cause of some work rather than a component of the system's competence.

On this unit's reading, each clinic is an activity system with an object, a community, rules, a division of labour and mediating artefacts, and the tool enters as a mediating artefact rather than as a replacement for the activity (Leont'ev, 1978; Engestrom, 2000). Clinic A's object is the waiting list; clinic B's object is the stability of chronic conditions across a long relationship with the same patients. The tool proposes a next appointment, which fits clinic A's object and cuts across clinic B's, because a medication review is a step that belongs to the treating relationship rather than to a queue. Engestrom's (2000) distinction between short-lived, goal-directed actions and durable, object-oriented activity systems is what predicts the asymmetry: the tool can fit every action in clinic B's workflow and still work against the object that gives that workflow its point. The abandonment is therefore not user resistance but a contradiction between the tool and the object, appearing as workaround, then as disuse. The analysis would be testable if the object were fixed before it was observed. The study should, first, have two analysts independently write down the object of each clinic from documents and interviews, before seeing the tool in use, and report whether their accounts agree; this answers Kaptelinin's (2005) charge that the object is underdefined by giving the concept an inter-rater test. Second, it should predict in advance which clinic will abandon the tool and which contradictions will surface, then observe for six months. Third, it should record tool openings, the workarounds staff invent, and where each proposal is overridden, since Fu (2025) shows that contradictions show up as invented tools and shifts in moderation rather than as open refusal. The result that counts for the framework is that the pre-specified object and the pre-specified contradiction predict the clinic that abandons the tool and the form the abandonment takes; the result that counts against is that both clinics behave alike, or that the object has to be rewritten after the observation to fit what happened. Until that study exists, the unit's conclusion stands: the object and the contradictions explain a great deal, but the explanation is a commitment rather than a measurement.

The designers have built the plan into the system as its control structure. They assume that a user's purpose can be specified in advance and that the specification is a sufficient context for interpreting everything the user does afterwards, which is precisely the assumption Suchman (1987) subjected to test. Her copier study recorded first-time users of an intelligent help system that tried to infer a plan from whatever action it could detect, and found the relation between intention and action too weak for the inference to hold: when help meant 'what happened?', the machine could answer 'describe the options of this display', and users misread the machine by imposing conventions from human conversation. Suchman named the resulting trouble the false alarm, where a user is led to believe an error occurred when none did, and the garden path, where an error goes unnoticed, and noted the system had no access to the user's situation to detect either. Applied to the clinic, the critique predicts that the visit-type list works while the case fits a listed type, and that the mandatory selection fails exactly on the complex cases where the user's problem cannot be named in advance; the departure from the expected path is then read as error rather than as the ordinary way work is done. That the failures cluster on the non-routine cases rather than being spread evenly is the signature Suchman's account predicts. Lave (1988) supplies the evidence that competence itself moves with the setting, since the same people who averaged 59 per cent on school-style tests were correct 98 per cent of the time on best-buy arithmetic in the supermarket. The designers' strongest reply is the symbolic one from Vera and Simon (1993): that symbolic systems already perform well in temporally demanding tasks and that the goals of situated action can be met within a symbolic framework, so the answer is a better plan model rather than no plan model. They might add that procedure models remain valid for routine work, which is where their system appears to succeed. The situated side answers, with Suchman (1993), that this reinstates the plan-based assumptions under dispute, and Anderson, Reder and Simon (1997) argue the two perspectives differ in form rather than substance and can be reconciled, so the exchange does not close on its own. What would settle it is a comparison that fixes the measure and separates the cases: ask whether requiring a stated visit type changes performance only on the non-routine cases while routine performance is untouched, and report whether any breakdown follows from the plan requirement itself rather than from the unfamiliarity of the case. Until such evidence exists, the unit's thesis stands, that the plan is an account produced within action, that the routine-task reply answers a narrower question, and that the dispute is unresolved.

The claim as written is not yet a claim, because it does not say which sense of embodiment it uses. Dourish (2001) defines embodiment as the way physical and social phenomena unfold in real time and real space, a sense broad enough that anything with a screen would qualify; Fishkin (2004) uses a narrower sense, an axis of his taxonomy alongside metaphor, on which tangibility is a spectrum rather than a binary property. The review should therefore require the studio to say which sense it means and where the design sits. If the claim is that the controls are intuitive, it takes on the burden that Hurtienne and Israel (2007) accept when they define intuitivity and propose image schemas and their metaphorical extensions as a taxonomy, and it must be measurable. The strongest available evidence is the comparison Antle, Corness and Droumeva (2009) built: the same environment twice, differing only in the embodied metaphor in the mapping layer. Their outcome was qualitative, about the complexities of designing intuitive interaction, so the studio should not expect a clean win, and should pre-specify what outcome the physical design is supposed to produce. Fishkin (2004) and Klemmer, Hartmann and Takayama (2006) supply the design vocabulary the report needs, including the five themes of thinking through doing, performance, visibility, risk and thick practice; the desk case is precisely about visibility and performance, since a fader's position is readable without looking while a screen route is not. The review should also require the studio to state the strongest counter-argument it faces. Clark (1997) holds minimal representationalism, keeping representations but making them local, personal and action-oriented, and Wilson (2002) shows that the six claims made under the heading of embodied cognition are unevenly supported, with the strong environmental claim deeply problematic. A documentation claim that survives should name one of those claims, name the result that would refute it, and report both.

The plan treats a method as a technique and is likely to produce what Gaver, Boucher, Pennington and Walker (2004) warned against. They report probes sent to twenty volunteer households and the fragmentary returns they produced, and they list the losses that follow from rationalising the approach: unambiguous questions return what the researcher already knows, summarising produces an average picture that fits nobody well, analyses blunt the designer's contact with the material, and the demand for justifiable accounts constrains imaginative engagement. Coding returns into a ranked list of needs does the summarising and the analysing at once, so the study will read the volunteers' lives through the team's existing categories. Boehner, Vertesi, Sengers and Dourish (2007) showed that this is the field's settled pattern rather than an aberration: probes travelled widely while the methodology behind them did not, which left underacknowledged disagreement about valid interpretation and about the relation between a method and its methodology. The assessment should therefore ask what conception of knowledge the plan assumes, since the answer decides whether the ranked list is a finding or a distortion. To describe what the service is part of, I would use the practice vocabulary of Shove, Pantzar and Watson (2012), who treat a practice as an integration of materials, competences and meanings held together in the doing, and who analyse how practices recruit or lose carriers. Cooking, sharing and shopping are practices with carriers, and the app is a candidate element in them, not a container of needs. That framing also changes what counts as success, because a service can change which practices people sustain rather than what they prefer. Accountability should then be reported on the field's own terms. Dourish (2006) argues that the criterion of implications for design misconstrues ethnographic inquiry and misses where it helps HCI, so the team should state its analytic commitments in advance rather than deriving them at the end. Bonuses would follow from Adamu's (2023) question about whose work the field reads and from his caution that a frame centred on race can be dismissive of other forms of recognition, since the volunteer sample and the coding scheme decide whose practice is described at all.

The pilot measures the technical subsystem and leaves the social subsystem unmeasured, which is the error Trist and Bamforth (1951) diagnosed in mechanised coal-getting: a method that split work across shifts and broke the intact groups produced higher absence and mutual blaming, and the promised productivity did not arrive. A simulated ward cannot show whether the handover system moves the exception path and the coordination work onto some other person, which is exactly what their coupling argument predicts. Clegg (2000) supplies the diagnostic question: joint optimisation of social and technical systems remains rare, most investment in information technology is technology-led, and users seldom have substantial influence on development, so a supplier-run technical pilot is the pattern he warns against, and his own verdict that principles are necessary but not sufficient should temper any claim that passing the pilot settles the design. Participation should include the people who hold the work together at shift change, not a representative sample of staff. Bjerknes and Bratteteig (1995) let me name the justification rather than assume it: participation can be argued for on quality, because the participants know the work, or on democracy, because the people whose work is being redesigned are owed a say, and the trust should state which it intends, since a quality argument can be satisfied by consulting a few experts while a democracy argument cannot. Bodker (1996) adds that participation is a negotiated arrangement whose conditions decide what it can achieve, so the plan must say what time, information and authority participants will actually have rather than treating a workshop as sufficient. I would hold the participation to the standard Frauenberger and colleagues (2015) propose, using their four lenses of epistemology, values, stakeholders and outcomes to make the choices explicit, while noting that they locate rigour in debate and reflection, and to the accountability Halskov and Hansen (2015) imply when they show a field spread across five main categories with participation defined in many ways. The test the trust should run is the comparison the tradition has not reported: introduce the system on two comparable wards, one designed through joint optimisation with the handover group and one adopted as supplied, and follow both past go-live, measuring not only error rates and task times but the frequency of unrecorded workarounds, the number of handovers that need a second person to reconstruct, and staff absence and reported strain in the weeks after the change. That is roughly Clegg's (2000) prediction made checkable, since he expects an unresolved design issue to produce a system that does not meet its objectives.

The assessment should begin by treating the bin, the classifier, the dashboard and the review queue as participants that carry action rather than as tools. Latour (1992) supplies the vocabulary: a rule about what may be recycled has been delegated to the classifier, and the requirement that residents sort correctly is prescribed back onto them through the bin's refusal and the review queue that catches the failure. The classification standard and the maintenance work of emptying, calibrating and repairing are participants in the sense Star (1999) means, since they are transparent while the system works and become visible when a bin jams or the dashboard misreports a fill level. Star and Griesemer (1989) add the sharper question of which artefact lets two communities cooperate: the item categories in the app may function as a boundary object between residents and the council only if both can use the same categories for their own purposes. The distribution of the standard is where Star (1990) bites, because a standard enforced by the bin presumes a single social world, and the residents whose packaging, cooking or shopping falls outside the coded categories are the marginal members who carry the manual review as uncompensated work. On the council's claim, a symmetrical description can report the arrangement and describe where the work has moved, which is a real gain, but it cannot rank the arrangement, and Callon (1999) explicitly defends the absence of a theory of the actor that would make such ranking easy. Latour (1992) declines the same question in his own seat-belt case. Puig de la Bellacasa (2011) supplies what the symmetric account omits, since matters of care require a commitment to the neglected things and people rather than an even-handed description of the network. The comparison the council should run is therefore between the automated system and its predecessor at one site, measuring the manual review requests per household, the time residents spend on them, the rejected-item rate for packaging that the coded categories do not cover, and the collection rounds that were reordered, followed long enough to see where the exception path settles. That is the evidence that would turn a description into a judgement.

The assessment should begin by separating three questions the board's brief runs together: what the framework is a theory of, what it is for, and what would count as evidence for it. On the first, Hornbaek and Oulasvirta (2017) show that even interaction itself is rarely defined directly, so the team should be asked what practice the framework describes and what it leaves as noise. On the second, Rogers (2012) records that theory in HCI has been used for testing, and also for describing, explaining, critiquing and generating designs, so 'theoretically grounded' names no particular function. Halverson's (2002) four attributes give the board a workable test, since the framework can be scored separately on descriptive, rhetorical, inferential and application power rather than accepted or rejected as a package. The team's claim that the framework beats usability testing implicitly asserts application power, that the theory tells designers what to build, which is exactly the claim Rogers (2004) found the imported theories largely failed to substantiate in practice. The contested question then sets out the rival answers: theory might make the setting intelligible and supply its own criteria (Harrison, Tatar and Sengers, 2007; McCarthy and Wright, 2004), or it might generate design through counterfactual thought experiments about what interaction would be like (Oulasvirta and Hornbaek, 2021), as Beaudouin-Lafon's (2000) interaction model does by defining a design space with properties for comparing techniques. The evidence the board should require follows from Halverson's scheme. Let the team name one design decision in the catalogue, state in advance what the framework predicts the change will do, and record whether the outcome matched; and additionally test whether the framework generated options the old process did not. A learned catalogue changes the target: the system's errors are unpredictable and damage experience in ways ordinary system errors do not (Yang et al., 2020), and design guidance for such systems has largely preceded any theory of them (Amershi et al., 2019), so the board should treat the framework's predictive claims as weaker, not stronger, in that setting.

A strong answer starts by deciding what the display is for. The score prompts a review; it does not make the decision. On that basis the display should not show a bare percentage, which implies a precision the model does not have and invites the clinician to compare 7 per cent with 9 per cent as if the difference meant something. A band is more honest, and it should be accompanied by the model's measured performance in this population and a short statement of the base rate, because Parasuraman and Riley (1997) show that ignoring the base rate is what produces both false-alarm fatigue and misplaced confidence. Show what should happen next rather than only the number: which observations to take, and how long to wait before reviewing. Where the model is least reliable, say so at the point of use, since Bainbridge (1983) explains that the unusual case is the one the clinician has least practice with and the one on which the tool is most likely to fail. The calibration test should measure whether the clinician's stated confidence tracks the model's actual reliability rather than whether the clinician agrees with the model. Use a set of retrospective cases with known outcomes, present them in a counterbalanced order, and record the decision and the stated confidence by model reliability band. Compare a point-estimate display against the banded display, and measure overreliance and underreliance separately. What the test cannot show is whether calibration survives a real shift on a busy ward, or whether skill decays over months, which would need a longitudinal design that a single study cannot supply.

The paragraph must name a criterion rather than assert fairness. A defensible version reads: the model ranks claims for review; it is calibrated so that a given score means the same risk in every group; we have chosen calibration within groups over equal error rates across groups; this choice means that, for a group with a higher underlying rate of the outcome, the same score produces a higher false negative rate for that group; we accept that trade because the review, not the model, decides the claim, and we report error rates by group in the model card. The justification should cite Kleinberg, Mullainathan and Raghavan (2017), whose result is that calibration within groups, balance for the negative class and balance for the positive class cannot hold simultaneously except in constrained cases. That result removes the option of claiming all three, so the team must choose and state the cost. The alternatives should be named and priced: equalising error rates across groups can require adjusting scores in a way that decouples the score from actual risk, which is defensible when the consequence of an error is severe and unacceptable, and harder to defend when the score drives only which claims are read first. The answer should also note what documentation cannot do. Mitchell and colleagues (2019) show that a model card records intended use and performance by group, and Gebru and colleagues (2021) show that a datasheet records how the data were collected, but neither prevents harm. The strongest answers add the operational safeguard that makes the choice survivable: human review with the capacity to overturn, and monitoring of the error rates the paragraph promises.

Allocate control by comparing the expected benefit of the model acting with the expected cost of it being wrong, and say who bears each cost, as Horvitz (1999) frames the decision, rather than by a level of autonomy such as automatic at night. The two costs fall on different people: a missed deterioration harms the patient, while a false alert spends the nurse's attention, and the weights belong to the parties who bear them. The proposal must be judged on the joint outcome, not on model accuracy. The nearest measurement is Hemmer and colleagues (2023), who found that AI-initiated delegation improved task performance and satisfaction in an experiment with 196 participants, and their point is that the human side is measured at all. Do not assume that result transfers: Adam and colleagues (2024) found that delegation the system initiates raised perceived self-threat and lowered willingness to accept, so a night mode that imposes handovers may meet a cost the performance figures do not show. Before enabling it, require a study that varies who initiates the handover while holding the interface constant and measures both the joint clinical performance and the nurses' acceptance, satisfaction and self-efficacy, which is the design the unit argues the field has not run. Disclose the limits before first use, since Kocielnik, Amershi and Bennett (2019) found that the same accuracy produced different perceptions and acceptance depending on which error the system was designed to avoid and how its limits were set. Finally, do not rely on asking the nurse when to attend, because Avrahami, Fogarty and Hudson (2007) found that self-reports of interruptibility diverged from observers' estimates of the same moments.

State first that 94 per cent on a held-out set is a claim about a distribution, not about the countries in which the feature will run, because competence is a property of the training distribution rather than the code (Domingos, 2012; Quinonero-Candela and colleagues, 2009). Treat the held-out set as suspect on two counts: Torralba and Efros (2011) showed that a benchmark partly measures its own test set, and Recht and colleagues (2019) found that rebuilding standard test sets by the original procedures lowered accuracy by 3 to 15 points on CIFAR-10 and 11 to 14 on ImageNet. Expect the failure where the training and deployment populations differ: the support centre in one country is a sampling frame, and Shankar and colleagues (2017) linked exactly such a frame to poor performance on under-represented regions. Note that this is a collection decision, and Sambasivan and colleagues (2021) found from 53 practitioner interviews that data work is devalued and its failures surface late and expensively. Before sign-off, require per-country performance on data from each deployment region, and a statement of what happened when training and deployment populations disagreed, which is the measurement Koh and colleagues (2021) built WILDS to force. Require a datasheet and a data statement for the training corpus, recording the composition, the collection process and the population it claims to represent (Gebru and colleagues, 2021; Bender and Friedman, 2018), and recognise with Sculley and colleagues (2015) that data dependencies are a source of technical debt that will change under the feature. Do not accept a model-side adaptation as a substitute for asking whether the training population resembles the users, since Csurka (2017) surveys adaptation methods without comparative success and treats the question as a modelling one.

The chart answers an engineer's question: it reports which of the model's inputs moved the score. It does not answer the candidate's question, which Miller (2019) shows is contrastive, selective and actionable: why this outcome rather than the one they expected, which few causes mattered, and what would change it. Liao, Gruen and Miller (2020) found that practitioners' needs centre on how the system behaves in general and what to do when it fails, which a ranked attribution list addresses poorly. Accuracy of the chart is not the defence the lead assumes. Adebayo and colleagues (2018) showed that some widely used saliency methods produce essentially unchanged maps when the model's parameters or labels are randomised, and Ghorbani, Abid and Zou (2019) showed that perceptually indistinguishable inputs with the same predicted label receive dramatically different maps, so I would require both sanity checks and a stability test on this model before citing accuracy. I would require evidence about what the feature actually does. Poursabzi-Sangdeh and colleagues (2021) found in pre-registered experiments with 3,800 participants that clearer, more compact models did not increase following and reduced detection of the model's large mistakes, so legibility is not the outcome being bought. Wang and Yin (2021) found that explanation properties dissociate, so I would not accept understanding scores as evidence of better decisions. The form I would test instead is the contrastive one. Bove and colleagues (2023) found that plural counterfactual examples with comparative features raised objective understanding and satisfaction over a single counterfactual with 112 participants, and VanNostrand and colleagues (2024) found region-based counterfactuals raised objective, subjective and confidence measures over point-based ones with 252 participants, so a route to action is designable and measurable. I would judge the shipped version on the candidate's decision and understanding, measured against that alternative, at the standard Doshi-Velez and Kim (2017) set for application-level evaluation. Finally, Bhatt and colleagues (2020) found most deployments use explanation to debug models rather than to inform affected people, so the lead's claim that this satisfies a duty to explain needs a purpose stated and tested, not asserted. If the duty is to give the candidate a usable reason, an inherently interpretable scoring rule of the kind Rudin (2019) argues for would be the stronger option, and it should be compared against the explained model rather than assumed away.

The proposed outcome is the wrong target. Lee and See (2004) argue that the design goal is appropriate reliance rather than the quantity of trust, and the empirical record shows why: Dzindolet and colleagues (2003) found that an explanation of why an aid might err raised trust and reliance even where that trust was unwarranted, and Zhang, Liao and Bellamy (2020) found that a confidence display improved trust calibration without improving joint accuracy. A rise in reported trust therefore tells me nothing about whether clinicians will accept correct alerts and override incorrect ones, and Schemmer and colleagues (2023) show those two behaviours must be measured separately or the effect is concealed. The automation literature also predicts a specific harm. Skitka, Mosier and Burdick (1999) found that participants with a very but not perfectly reliable aid performed worse on a monitoring task than those without it, and made omission errors when the aid did not prompt them; Parasuraman and Manzey (2010) report that this bias appears in expert as well as naive participants and is not removed by training or instructions; and Bainbridge (1983) adds that vigilance to a source on which little happens cannot be sustained for long, while unused skills deteriorate. Before the alert ships I would require the two-dimensional reliance measure of Schemmer and colleagues (2023), a comparison of interface variants with the model held fixed, and an estimate of the effect per level of clinical expertise, because Tschandl and colleagues (2020) found that AI support improved accuracy over either party alone and that the least experienced clinicians gained most, while faulty AI misled experts too. For judging the shipped alert I would require joint decision quality and the rate of overridden correct alerts, reported separately for the least and most expert groups, and, following Bansal and colleagues (2021), a statement of whether the alert threshold was tuned for the team's outcome rather than for alert accuracy alone.

The single number is the first thing to question, and Guo and colleagues (2017) explain why the figure is not yet what it appears to be: the networks they examined were poorly calibrated and overstated their correctness, so a score becomes a usable probability only after recalibration. The recalibration does not travel either. Ovadia and colleagues (2019) found that temperature scaling achieved low expected calibration error at low dataset shift, but that the error increased significantly as shift increased, so a number calibrated on an in-distribution validation set is not guaranteed valid when the scanner, the referral population or the coding practice changes. The department should therefore state which distribution the number is calibrated on and monitor shift, and Angelopoulos and Bates (2023) show what the stronger alternative looks like, since a conformal prediction set carries a coverage guarantee in advance rather than a point estimate. The representation matters as much as the quantity. Gigerenzer and Hoffrage (1995) found that statistically naive participants derived up to 50 per cent of their inferences by Bayesian algorithms in frequency formats, and Cao, Liu and Huang (2024) found that calibrated uncertainty presented alone was inadequate in a screening task while a frequency format allowed users to adjust their reliance. I would ask for the decision the interface must support to be named first, since Fernandes and colleagues (2018) showed that decision quality moved with the representation, dotplots reaching 97 per cent of optimal payoffs five percentage points above control, while the same uncertainty stated in words performed worst. Ancker and colleagues (2006) add that reasoning accuracy, behaviour and preference are separate outcomes, so the department cannot infer one from another. The confidence figure also acts on the reader, not only on the case: Li and colleagues (2025) found that self-confidence aligned with the AI's confidence, persisted after the AI was withdrawn, and impaired calibration for most participants, so showing one number per study may erode the radiologist's own sense of certainty. For judgement I would fix the quantity, vary the representation, and score clinical decision quality and the override rate, reporting effects by experience and by statistical familiarity, because the audience position in this unit, from Spiegelhalter, Pearson and Short (2011), Cao, Liu and Huang (2024) and Ancker and colleagues (2006), is that no single format serves every reader equally. Finally, the department should expect that disclosing uncertainty is affordable: van der Bles and colleagues (2020) found across five experiments with 5,633 participants that trust fell only slightly, and mostly where uncertainty was communicated verbally. The display also carries an access condition. A chart is read only by a person who can see it, so the graphic should be published beside a textual equivalent of the same quantity, and Duke (2023) supplies the evidence that a numerical range in the text improves how a reader interprets a probability. A ward on a slow connection is the same case in another form, because the richest display is the one that takes longest to load.

The single figure is a defensible choice, but it is a choice, and the review should begin there. Kleinberg, Mullainathan and Raghavan (2017) prove that calibration within groups, balance of error rates across the positive and negative classes, and balance for the positive class cannot hold at once except in highly constrained cases, so demographic parity is buying one property at the cost of others. Chouldechova (2017) makes the practical consequence explicit: where the prevalence of the outcome differs between groups, the criteria cannot all hold, and the reported figure must be read alongside the base rates. Naming the errors follows directly. Demographic parity constrains outcomes rather than the influence of protected characteristics, which is the first of the two families Corbett-Davies and colleagues (2018) describe, and they argue that such constraints typically produce policies Pareto dominated by a simple threshold rule and can harm the groups they were written to protect. The review should therefore require the team to state which errors parity assigns to identifiable applicants, and to compare it against at least one alternative criterion, such as the equalised odds that Hardt, Price and Srebro (2016) define and can enforce by post-processing. The target variable needs the same scrutiny, because Obermeyer and colleagues (2019) found a commercial algorithm that predicted cost rather than illness, with the result that Black patients at a given score were considerably sicker, and correcting the disparity would have raised the share receiving additional help from 17.7 to 46.5 per cent. 'Risk of long-term homelessness' must be defined before it is predicted, and the definition must be tested for that kind of proxy failure. Reporting must then be disaggregated, because Buolamwini and Gebru (2018) found average performance concealing error rates of 34.7 per cent for darker-skinned females against 0.8 per cent for lighter-skinned males, and the assumptions behind the measurement must be stated, as Friedler, Scheidegger and Venkatasubramanian (2021) and Jacobs and Wallach (2019) require, if the claim is to be testable at all. Pagano and colleagues (2023) add that fairness metrics do not agree on a single use case, so the population and the metric must be named together, and Caton and Haas (2024) show that the literature supplies no standard for the team to inherit. If the council is in the United Kingdom or the European Union, Wachter, Mittelstadt and Russell (2021) indicate that legal review stays with people, since contextual equality is not reducible to a metric. Evaluation should hold the model fixed, vary the constraint, and measure accuracy together with harm to the affected groups, and the council should publish the result: Raji and Buolamwini (2019) found that all three targeted companies released new API versions within seven months of an audit, but the companies the audit did not name still lagged well behind, so disclosure works better as a structural duty than as a voluntary gesture.

The model card is a real contribution and the memo should say so before qualifying it. Mitchell and colleagues (2019) designed the card to record intended use, evaluation conditions, subgroup performance and known limitations, so the first question is whether this card states what the flag is for, how its threshold trades false negatives against false positives, and how it performs across the groups it will score. A card that reports one accuracy figure has not done the job it was designed for. The second question is audience. Arnold and colleagues (2019) write FactSheets for examination by consumers, meaning buying organisations, and Article 13 of the Artificial Intelligence Act (2024) directs a high-risk system's transparency at the deployer rather than at the person affected. The university is the deployer here, and the students are the affected people, so the documentation as described is addressed to the institution and not to them. The team's claim of a 'right to an explanation' should be corrected rather than repeated. Wachter, Mittelstadt and Floridi (2017) conclude that the GDPR creates only a limited right to be informed about automated decision making and the logic involved, and warn that Article 22 is ambiguous enough to leave the protection toothless, while Selbst and Powles (2017) read the same articles as supporting a right to explanation to be interpreted functionally. Asserting the right as settled misrepresents a live contest. The claim that the system is audited also needs to be made specific. Raji and colleagues (2020) describe internal auditing as five lifecycle stages, scoping, mapping, artefact collection, testing and reflection, each producing documents assessed against the organisation's stated values, and Mokander and Floridi (2021) add that the process must be continuous and constructive and aligned with incentives rather than a one-off form. The memo should ask which stages were run, who set the values being audited against, and when the audit will be repeated. Finally, testing whether any of it works requires a comparison the literature has not run: systems that document and audit against otherwise comparable systems that do not, holding the task and the model constant, and measuring what the affected students experience, including how many are wrongly flagged and how the process is perceived. Without that comparison the university can report its documents but not its accountability.

The release note offers three claims and none of them carries the weight the team puts on it. Start with the benchmark score, because that is where the substitution is easiest to see. A BLEU score computed on a public summarisation benchmark compares the assistant's output with a reference string drawn from one distribution of items, which is the unit's definition of the problem: the score is portable, the sample is not. Reiter (2018) reviewed 284 correlations across 34 papers and concluded that the evidence supports BLEU for diagnostic machine translation evaluation and not outside it, so a summarisation figure is being used in a task and a population it was never validated for. Sai and colleagues (2022) show how many metrics exist and how thin the validation of each is, which is why the choice of one metric should be justified rather than selected. The claim needs two things before it means anything: what population the benchmark items were drawn from, and how the score behaves on text from the customers' own meetings. The 'state of the art' label is the construct-validity claim Raji and colleagues (2021) warn about, since performance on a benchmark is being read as evidence of progress towards a goal the benchmark was not designed to measure. Ask what it was state of the art on, and against which other systems, and note that Bowman and Dahl (2021) find most benchmarks failing the criteria a benchmark should meet, because unreliable systems score so highly that improvement cannot be shown. The internal preference study is the most informative of the three and the least documented. Human judgement is a measuring instrument, and Clark and colleagues (2021) found untrained evaluators at chance when distinguishing machine- from human-authored text, with training gains of up to 55 per cent that were not significant across domains; van der Lee and colleagues (2021) add that published human evaluations vary so much in design, recruitment, instructions and reporting that a bare statement of preference is not interpretable. So the review should demand the design: which raters, on which items, under which instructions, and what they preferred on dimensions that matter for a meeting. Chiang and colleagues (2024) show that preference can carry a ranking at scale, but ranking models on open prompts is not certifying competence on a named task. What the review should require before release is an evaluation that names the deployment setting, holds the task and the items constant, reports performance on meetings from the customer's own domain, and states what the team did not test. That is a comparison the literature this unit reviews has not run, and its absence is the strongest reason to withhold the word proof.

The review should separate the three claims and test each against what the unit's sources actually support. Take verifiability first, because it is the claim most likely to be mistaken for a property of the interface. A citation is evidence only if someone has checked that the cited passage says what the sentence claims, and Liu, Zhang and Liang (2023) show how far deployed systems fall short: auditing four commercial generative search engines, they found on average 51.5 per cent of generated sentences fully supported by their citations and only 74.5 per cent of citations supporting the associated sentence. The team should therefore report a measured attribution rate rather than assert verifiability, and it should use a defined standard, such as the Attributable to Identified Sources framework of Rashkin and colleagues (2023), which requires each output about the external world to be verified against an independent provided source and separates attribution from fluency. The grounding itself is the architectural move of Lewis and colleagues (2020), whose retrieval-augmented generation improved factuality and made provenance inspectable, and of Gao and colleagues (2023), whose RARR post-edits unsupported content; both are gains, and neither establishes that a reader can now tell a supported sentence from an unsupported one. The transparency claim is weaker still. Wei and colleagues (2022) established that chain-of-thought prompting improves reasoning above a scale threshold, but Turpin and colleagues (2023) show the stated reasoning can systematically omit the feature that changed the answer, with accuracy falling by as much as 36 per cent across thirteen tasks when the model is biased toward wrong answers. A reasoning panel is a persuasive account, not a record of the computation, so the team must not present it as one. The red-team claim is the cleanest substitution: Greshake and colleagues (2023) demonstrated indirect prompt injection through retrieved content against real systems and reported that effective mitigations were lacking, and Zou and colleagues (2023) showed that automated adversarial search bypasses alignment and transfers between models. A red-team report that finds nothing is a statement about the search that was run, not an estimate of safety, so the team should report the attacker's strength and the number and kind of attempts, following the scaling comparison of Ganguli and colleagues (2022). The evaluation I would require holds the task and the document set fixed, reports the unsupported-statement rate and the citation precision before and after grounding, measures attribution with a stated standard, tests the retrieval path against injected instructions of stated strength, and states what was not tested. Finally, the answer should note that the assistant's own evaluation is now often done by a language model judge, and that Wang and colleagues (2024) show such a judge carries a positional bias, so any automated scoring must be calibrated against human labels.

The figures do not establish complementarity, because the comparison that matters is missing. A combination is complementary only when it beats the better of the two parties working alone (Vaccaro, Almaatouq and Malone, 2024), and here the better party is the model at 0.88 while the combination reaches 0.86. The pilot therefore shows a small loss against the strongest arm, which is the pattern the meta-analysis reports on average. What the figures cannot show is why. The design shows the model's suggestion first and asks for confirmation, which is the arrangement Lai and Tan (2019) found raises accuracy while transferring influence from the person to the machine, and Bansal and colleagues (2021) found raises acceptance of a recommendation whether or not it is correct. Confirmation after the answer is visible is assent, not a second independent judgement, so the two error structures are unlikely to differ in the way complementarity needs (Steyvers and colleagues, 2022). The design change is to obtain the radiologist's own read before the model is shown, keep the model's confidence available, and route cases on a stated rule, which is what learning to defer does when it accounts for the human's errors and the cost of deferring (Madras, Pitassi and Zemel, 2017) and what training on the team's outcome rather than the model's accuracy is for (Wilder, Horvitz and Kamar, 2020). The next evaluation should be pre-registered, fix the case set and the interface, and compare four arms: model alone, radiologist alone, the current design and the independent-read design. It should report joint accuracy against the better party with a confidence interval, the calibration of both parties' confidence signals, and a measure of retained human knowledge, following the convergence result of Fugener and colleagues (2021). Watch the case mix too, since assistance helped on eighteen tasks inside the model's capability and harmed on one outside it in Dell'Acqua and colleagues (2023). I would recommend against deployment if the independent-read arm still fails to exceed the model alone on a powered sample, or if the model's errors are concentrated on the rare findings where the radiologist's judgement is the only defence.

The two goals pull in different directions, and a strong answer separates them rather than averaging. For exploring a building that no longer exists, a fully virtual environment is defensible, because the real gallery contributes nothing to that task and would only interfere. The visitor is being asked to experience a place, so the thresholds that matter are the system's immersion: tracking that keeps the view locked to the head, a field of view wide enough to avoid a tunnel, and low motion-to-photon latency so the world does not lag behind the turn. For seeing the real objects that are present, augmented reality is the better position, because the real environment carries information the system cannot reproduce, namely the actual physical exhibits. Here the binding constraint is registration, the alignment of virtual content with the real scene, since a label or overlay that drifts a few millimetres will point at the wrong object and destroy trust. Unless a serious design reason emerges, avoid stereo depth where it is not needed, because a display that forces vergence and accommodation apart will fatigue visitors over a long day, and avoid controller based navigation in favour of walking that matches the body, since sickness risk rises with exposure time and mismatched locomotion. To know whether the choices worked, measure behaviour rather than opinion. Log where visitors turn and how long they look, record whether they physically reach for virtual objects, and note disorientation and dropout rates as sickness evidence. A questionnaire is useful as a secondary check, not as the primary claim.

The critique should attack the inference, not the enthusiasm. A forty per cent error reduction is a claim about transfer, which means the skill must survive outside the headset, in the real task, at a later time. A laboratory study with two short sessions measures performance on the trained task in the training setting, which is the weakest possible test of transfer. The presence questionnaire is the second problem. Presence is a subjective correlate of immersion, and self-report tells you what participants believe about the experience, not how they responded, so a high score is evidence about the session and not about the training benefit. The evaluation also omits the measures that would make the claim checkable: what counts as an error, who judged it, whether the comparison group did the same task another way, and how long sickness lasted. A better study compares the virtual training against the current method, randomises staff to the two conditions, and measures outcome on the real task at a delay of weeks rather than minutes. It reports error rates with an error definition fixed in advance, tracks dropout attributable to simulator sickness, and uses behavioural or physiological evidence alongside any questionnaire. Even then the design would not establish that the effect generalises to other sites, other equipment or other staff, and it would not show which part of the virtual system caused the gain. That last limit is the one to state plainly, because it decides whether the result can be replicated elsewhere.

The report confuses a measure with the experience it is meant to indicate. Immersion is a property of the system, so a larger field of view and stereo rendering are checkable facts about the build; presence is an inference about a person, and the questionnaire is a self-report response (Slater and Wilbur, 1997). Higher scores therefore show that participants reported more presence in one version, which is a real finding about responses, but they do not establish that the training experience improved, because no one has measured training. The strongest available quantitative support for the studio's implicit assumption is Cummings and Bailenson (2015), who found a medium-sized effect of immersive technology on reported presence, with field of view and stereoscopic visuals among the more impactful features, so the manipulation is the right kind. The difficulty is the comparison itself. Usoh and colleagues (2000) found that presence questionnaires could not reliably separate a real office from a virtual replica, which is a cross-environment comparison, and Skarbez, Brooks and Whitton (2021) found that existing questionnaires could not separate the effects of immersion from those of coherence, while heart rate tracked coherence rather than immersion. A confound is therefore likely: the newer version may look more credible or less broken rather than more immersive, and the order of conditions, the content and the instrument version all need reporting. The study that would settle it is a within-subjects comparison that manipulates the two system features while holding content and coherence constant, uses a named instrument, and adds a behavioural or physiological measure taken in the same participants, following the logic of Meehan and colleagues (2002) and Bahavan and colleagues (2023). If the presence difference survives that design and predicts a training outcome, the studio has a case; if it does not, the honest conclusion is a difference in responses to a comparison (Skarbez, Brooks and Whitton, 2017).

The report establishes that trainees responded differently to the new version; it does not establish that they learned more. Presence and motivation are the variables that immersion reliably moves, and Makransky and Lilleholt (2018) show that these are exactly the measures on which an immersive lesson scores highest, with the largest differences on presence and motivation rather than on learning. An immediate recall test is a weak outcome on its own, because it is the measure most likely to reflect novelty and attention rather than durable understanding, and the unit's sources show that an immediate advantage does not always survive: Makransky, Terkildsen and Mayer (2019) found more presence and less learning in the same participants, and Parong and Mayer (2020) found worse transfer with more extraneous cognitive load. The comparison is also unfair as it stands. Hamilton and colleagues (2020) report that the pooled advantage of immersive lessons is largest when the comparison is a conventional lecture rather than an equivalent non-immersive activity, so a head-mounted lesson against a desktop slideshow is the favourable comparison that flatters the result. The confounds to check are the content and duration of the two lessons, which should be matched; the order of conditions and any novelty effect; whether the immediate test was followed by a delayed test; the participants' prior experience with headsets; and sickness and dropouts, which would remove the trainees least able to tolerate the medium. The design that would settle the claim holds the instructional method constant across the two conditions so that only the medium varies, measures transfer rather than recognition, measures cognitive load and affect in the same participants, tests again after a delay, and reports sickness and dropouts. This is the design Makransky and Mayer (2022) approach and Petersen and colleagues (2023) test from the instructional side, and no study yet meets all of its conditions at once.

The data sheet answers the wrong question. For an overlay registered onto a real machine, the binding constraints are the accuracy of the tracker and the end-to-end latency of the pipeline, not the panel. Azuma (1993) is the starting point: in augmented reality the virtual objects supplement rather than replace the real world, and even tiny registration errors are detected at once, so a tracker must be accurate to a small fraction of a degree in orientation and a few millimetres in position. He gives the arithmetic to make this concrete for a maintenance task, where an angular error of 1.5 degrees in head orientation moves a virtual cup two metres away by about 52 mm. An arrow pointing at the wrong component is exactly that failure. The second number to establish is the combined latency of tracker and graphics engine, which Azuma defines end to end rather than as a display property, and which Steed's (2008) pendulum method measures on the assembled system with an ordinary video camera. The threshold to compare it against is not one figure. Mania and colleagues (2004) and Ellis and colleagues (2004) report that the head-tracking just noticeable difference averages about 15 ms or less and that perceptual stability requires latencies below 16 ms, while Azuma (1993) and Jerald and Whitton (2009) make the requirement depend on how fast the head turns, so a technician who turns quickly to a new part is holding the system to a stricter budget than one who holds still. Because the task requires a wide view of a large machine, field of view is also at stake, and Lin and colleagues (2002) show it is a trade: presence and sickness both rose with the field and levelled off beyond 140 degrees. The test I would run therefore holds the task constant and varies the budgets: measure tracker resolution and end-to-end latency on the delivered unit, have technicians carry out a representative repair with the overlay, and record registration error at the component, time to complete, errors corrected, and simulator sickness, repeating the session on a second day to expose any novelty effect and logging dropouts. If latency and registration meet the perceptual figures above under the motion the task actually produces, the system is worth adopting; if they do not, no improvement in resolution or colour will rescue it.

The sheet answers the wrong question, because the depth a stereo display produces does not rest on resolution. A conventional stereo display holds the lens focused at the screen while disparity draws the eyes to the depicted distance, so the two cues disagree, and Watt and colleagues (2005) showed that this changes perceived depth rather than only tiring the eye. For anatomy review, where the trainee must judge how far a structure sits behind another, that is a perceptual error, not a cosmetic one. The conflict also carries a measured cost: Hoffman and colleagues (2008) found that observers spotted a stereoscopic target sooner, resolved depth more sharply and reported less distortion and fatigue when the display supplied near-correct focus cues, which turns the comfort complaint into a performance deficit. Before relying on the display I would establish the thresholds. Lambooij and colleagues (2009) report that discomfort climbs steeply past roughly one degree of disparity, and Shibata and colleagues (2011) derived a zone of comfort whose boundary depends on the viewing distance and on which side of the screen the content sits, so the safe region is conditional rather than one disparity figure. I would also note the limit of the optical remedy: Akeley and colleagues (2004) built a prototype that presents near-correct focus cues, and Ravikumar and colleagues (2011) turned it into a plane spacing and count, but MacKenzie and colleagues (2010) found that accommodation followed the simulated distance accurately only for image-plane separations up to about one diopter and broke down beyond that, which bounds how completely a multi-plane display can remove the conflict. The evidence also shows the depth error has more than one cause. Kelly (2023) confirmed the underperception of distance and found judged distance associated with field of view and resolution, and Krajancich and colleagues (2020) improved depth perception with gaze-contingent rendering without touching focus cues. The test I would run holds the task constant and measures outcomes rather than specifications: have trainees judge the relative depth of structures against a real model, record the disparity of the content, the exposure time, and the discomfort score, include a corrective or multi-plane condition where available, and repeat the session to separate novelty from capability.

The number 28 is not a rate and not a property of the training alone. Saredakis and colleagues (2020) pooled Simulator Sickness Questionnaire scores across 55 studies and found an overall total of 28.00, but gaming content scored 34.26, so a figure of 28 identifies the content that produced it as much as it identifies the employer's programme. The employer should therefore state the content, the locomotion method and the exposure length alongside the score, because the review found all three moved it. The single figure also hides how it was taken. Chang, Kim and Yoo (2020) report that most questionnaires are administered after the experience and so cannot describe discomfort in real time, and that users differ in which symptom they treat as the point at which they report sickness. A total collected at the end will therefore flatten a time course that may be rising, and it will miss anyone who stopped early. The employer should record dropouts and any reduction of exposure time, because a lower average can mean a programme people leave rather than one that is comfortable. That the number is 28 on the simulator questionnaire is a further problem, since Stanney, Kennedy and Drexler (1997) found the cybersickness symptom profile differs from the simulator profile, and Kim and colleagues (2018) had to retain only nine items in two components for head-mounted displays. The score is on a scale that the field itself has not settled, so it cannot support a claim about the whole workforce. Individual variation is the largest omission. Howard and Van Zandt (2021) found that motion sickness susceptibility, gender, prior experience and other differences all relate significantly to sickness and frame the unequal effect as a fairness problem, and Stanney, Fidopiastis and Foster (2020) found an unfitted interpupillary distance was a primary driver of the gender difference, so correct fitting partly removes it. 'Comfortable for everyone' is exactly the claim the average cannot make. A defensible plan would take one instrument plus one objective marker, such as the physiological signals Dennison, Wisti and D'Zmura (2016) recorded during exposure, administer them during the session rather than only at the end, report the distribution and the dropout count rather than a single mean, and check that the fitting and the content are held constant across the groups being compared.

There is no recommendation to apply, because no technique wins on every axis. Redirected walking preserves spatial knowledge best, since Langbehn and colleagues (2018) found it gave the best acquisition of spatial knowledge while it and teleportation were preferred over the joystick, which caused significantly more sickness. Because the task requires remembering the layout, that pushes the choice towards redirected walking or teleportation rather than a joystick. But redirected walking needs space and attention. Steinicke and colleagues (2010) report that a circular arc of radius greater than 22 metres felt straight, and Grechkin and colleagues (2016) later estimated the detectable radius at 11.6 or 6.4 metres depending on the estimation method, so in a 4 metre room the manipulation will be visible and the studio must either accept that or design around it, as Suma and colleagues (2012) did with overlapping rooms that stayed undetected at up to 31 per cent overlap in a 9.14 metre square workspace. Bruder, Lubos and Steinicke (2015) add that redirection consumes working memory, which matters in a session where the trainee is also learning. Teleportation avoids the space problem: Bozgeyikli and colleagues (2016) found point and teleport fun and user friendly, with no visible translational motion to provoke sickness, but it breaks continuous travel, and Prithul, Adhanom and Folmer (2021) note the questions still open about its effects. I would choose teleportation for the first roll-out, accepting reduced presence and broken travel in exchange for sickness control in a small room, and test it against redirected walking before committing. To settle the question, run one within-subject study with the same trainees and the same aisle-memory task, comparing redirected walking, teleportation and a joystick on presence, sickness, spatial knowledge and mental workload, and record the space each demands as a cost rather than controlling it away. Dylong and Nickel (2025) show why workload belongs in that battery: in their comparison of point-teleport and directed-steering variants, workload differed between the variants and one technique impaired performance mainly through visually induced sickness. Boletsis (2017) shows the field lacks a common outcome set, so the studio should publish its measures alongside its result.

The unit's evidence does not support a single technique for both halves of the task. The starting point is that pointing has a price set by the geometry of the task, since Fitts (1954) found movement time rising linearly with an index of difficulty built from amplitude and target width, and Lubos, Bruder and Steinicke (2014) apply that measure to head-mounted selection with the error rate held near four per cent. The first consequence is that the team cannot buy accuracy with naturalness. For the single point selections, where a small bolt must be chosen from a cluster, the evidence points away from a bare finger. Forlines, Wigdor, Shen and Balakrishnan (2007) found direct touch committed more selection errors than two mice, 18.9 per cent against 9.7 per cent, and judged the mouse better for single point work; Masurovsky and colleagues (2020) found a handheld controller ahead of controller free hand tracking on accuracy, speed and preference, with about seven millimetres of error against sixteen. If motor precision is the requirement, the team should either use a controller or use a refinement technique, since Bacim, Kopper and Bowman (2013) made selection much more accurate than ray casting by refusing to demand precision at any step, and Kruger and colleagues (2024) continue that line with score based selection. For the two-handed part, where a part is held while a tool is rotated, direct two-handed interaction has the stronger case, because Hinckley and colleagues (1994) argued that physical props let two hands express a spatial relation as one cognitive chunk, and Forlines and colleagues (2007) found direct touch beat two mice on their symmetric bimanual task and was preferred by ten of twelve participants. The trade must be stated: direct two-handed work costs shoulder endurance, which Hincapie-Ramos and colleagues (2014) show is predictable from posture and least when the arm is bent on the plane midway between shoulder and waist, so the team should specify that working plane. A controller would move the cost to the wrist instead. One further limit belongs in the brief. Lubos and colleagues (2014) found errors concentrated along the view axis and concluded that perception rather than motor control dominates, so the layout should keep small targets close to eye level and use a tolerant cursor rather than assuming a fingertip will land where it looks. Cockburn and McKenzie (2002) add a caution about claiming spatial memory benefits, since the third dimension made retrieval slower in their physical models and no statistically reliable advantage appeared among their virtual arrangements. To decide rather than demonstrate, the manufacturer should run one within-subject study with technicians performing both halves of the task under direct, mid-air and indirect selection, reporting effective throughput in bits per second alongside consumed endurance over sessions of the full thirty minutes, and testing component recall afterwards. No study in this unit's reading list runs that design, since the pointing experiments use short blocks of trials and the endurance work holds static postures.

An answer should begin from the definition and the constraint it contains. Azuma (1997) requires that the system be registered in three dimensions and names registration and sensing errors among the biggest problems, so the first thing to measure is the alignment error of the guidance, not the brightness or resolution of the overlay. Second, the team must handle occlusion. Wloka and Anderson (1995) showed that overlaying graphics on a real scene fails in three dimensions when occlusion between real and virtual objects is not addressed, so an annotation must be hidden by nearer parts and must hide the parts it covers. Third, it must test depth perception, because Swan and colleagues (2006) report systematic error in the perceived depth of virtual objects in optical see-through augmented reality, which means an arrow drawn behind a component may be seen in the wrong place. The evaluation should vary the variable the technical position names while holding the rest fixed: measure registration error and occlusion handling, vary only brightness and field of view, and report whether the task outcome changes. Speed alone is not enough, since Baird and Barfield (1999) found a see-through display produced the fastest assembly while leaving usability problems. Henderson and Feiner (2011) show the value of a within-subject comparison that includes an untracked display, which isolates the contribution of alignment. Finally, Howard and Davis (2023) found that head-mounted displays produced smaller effects than smartphones and tablets, so the team should test whether a tablet delivers the same benefit before committing to the headset.

Begin by refusing the vague claim. Marshall (2007) found that the tangible-interface frameworks mapped technical possibilities while giving little guidance on cognitive or social effects, and the evidence for learning benefits was thin, so 'more intuitive' is not yet a testable proposition. Fishkin (2004) shows why the team's proposed comparison is the wrong one: tangibility is a spectrum along the axes of metaphor and embodiment, so the first task is to name the property the tangible version changes and state a prediction about it, not to run a global tangible-versus-graphical trial. The existing evidence is mixed and mostly about outcomes other than performance. Horn and colleagues (2009) found that a tangible and a graphical exhibit were equally easy to understand, with the tangible version drawing more participation; Zuckerman and Gal-Oz (2013) found most users preferred the tangible modelling system even though performance was equivalent and it was rated worse on usability; and Manches, O'Malley and Benford (2010) traced a physical advantage to the properties of the representation rather than to physicality itself. Hornecker (2011) adds that much of the claimed benefit may come from social and spatial embedding rather than the material. The evaluation should therefore hold the task and information fixed, vary the degree of tangibility along Fishkin's axes, include the existing graphical tool as the control, and, following Hornecker and Buur (2006), control the social setting so collaborative use does not confound the result. Performance and preference must be reported separately, because Zuckerman and Gal-Oz (2013) show they can diverge. If the goal is engagement, a preference gain is a legitimate reason to ship; if the goal is performance, the evidence does not support shipping on the intuition alone. Adding a controlled study would help fill the gap Shaer and Hornecker (2010) describe.

The team is trading on Wilson's (2002) weaker claims and reaching for the strongest one. That people off-load work onto the environment, or act to simplify a problem, is at least partially supported; that an interface is part of the learner's cognitive system is the claim Wilson calls deeply problematic, and the evidence the team offers does not touch it. The prototype and the study are mechanism evidence. Kirsh and Maglio (1994) showed that people use actions to change a problem rather than to implement a plan, and Hartmann and colleagues (2006) treat a prototype as a hypothesis made testable, so a working tilt-and-shake toy can establish that the sensor distinguishes the states the design requires and that the behaviour is legible. The study with 39 children, modelled on Bakker, Antle and van den Hoven (2011), is a feasibility result: every child could complete the task, which says nothing about learning against an alternative. The strongest nearby evidence, from Hurtienne and colleagues (2010), shows that 92 per cent of participants produced metaphor-congruent gestures, but that tests whether a mapping is natural to the body, not whether it teaches. Before the benefit claim could be accepted the team would need a controlled comparison with a graphical version of the same task, learning pre-registered as the outcome rather than preference, and a report of the construct Kilteni, Groten and Slater (2012) separate into self-location, agency and ownership. Portions of this answer are synthesised from the cited sources rather than quoted from any single one.

A strong answer starts from the finding that timing matters more than content or preference. Iqbal and Bailey showed that deferring notifications to task breakpoints reduced frustration and reaction time compared with immediate delivery, at the cost of a deferral of about a minute and a half, and that content relevance decided which breakpoint suited a message. So the strategy should batch low-urgency updates to a natural pause rather than fire them on arrival. The counter-intuitive move to defend is that an important alert is not therefore best sent at once. Emergency content, such as a safety-interlock warning, still needs immediate delivery, and the answer should say why the breakpoint rule does not apply to it. The energy evidence supports batching for a second reason. Balasubramanian and colleagues found that frequent small transfers pay the radio's tail energy repeatedly, so coalescing updates saves power as well as attention. A good answer then designs the test. It would log when the app is in use and where pauses fall, then compare immediate against breakpoint delivery on reaction time, error and frustration in a field trial, reporting attrition. The thing that cannot be predicted from the evidence given is the size of the benefit for this specific population, because the original studies used different tasks and settings. Naming that gap earns marks; asserting that the effect simply transfers does not.

The privacy analysis should begin with Beresford and Stajano's definition of location privacy as the ability to prevent others learning one's current or past location, and with their point that a change of scale is a change of kind. Knowing where someone is now differs from a second-by-second history at sub-metre accuracy, so the answer should ask what is retained, for how long, and who can join it to other data. Their mix-zone idea, which unlinks identities through changing pseudonyms, suggests that sharing a position need not mean sharing an identity, and that the useful questions concern the whole flow rather than a single access check. The answer should note that controlling who queries a coordinate does not prevent that coordinate being joined later, so purpose limitation and retention belong in the design. The deployment half should treat the home setting as Brush and colleagues did. They found that adoption barriers were cost, inflexibility, manageability and security, and that installation, not sensing, is what loses households, so volunteers recruited through a laboratory link are not representative. The plan must state how consent is obtained in situ, including for people in the home who did not agree to be measured, how logs are minimised and stored, how participants may withdraw, and how attrition is reported rather than hidden. The strongest answers separate legal duties from design choices.

Background operation is not the same as calm, and the brief confuses the two. Weiser and Brown (1997) define calm technology by attention rather than by how a system is run: it engages both the centre and the periphery of attention and moves between them, and the periphery informs without overburdening. A system that runs invisibly but interrupts at moments of its own choosing has moved from the periphery to the centre at exactly the time the user did not ask for it, which is the opposite of the definition. What the team is really specifying is proactivity, and Rogers (2006) shows that proactivity is the goal the field fell short on: a system that acts on inferred context takes decisions on the person's behalf, and the more it does, the more often it must check whether it acted correctly, and every check is a demand for attention. The brief should therefore state when the application may speak, not only that it runs in the background. The team should also recognise how the goal has drifted in the literature: Satyanarayanan (2001) replaced Weiser's complete disappearance with minimal user distraction and conceded that complete invisibility may be impossible, so a single well-timed interruption is not automatically a failure. What settles the claim is measurement in the setting where the application is used rather than a laboratory that holds attention, which is the direction Ntoa (2024) argues evaluation must take, and Ntoa's review of 46 frameworks shows that most specify no metrics at all, so the team cannot assume an off-the-shelf instrument exists. Chalmers (2003) adds a practical test that the team should adopt: if the application depends on location or connectivity, the seams will show, and the design should say what the user sees when the inference is wrong rather than promise a service that never breaks.

The claim that the app will understand context is doing work no definition can support. Dey (2001) defines context as any information that can be used to characterise the situation of an entity, and this unit's argument is that such breadth is useful precisely because it refuses to rank signals; it cannot tell the team whether walking speed, calendar entries, time of day or the location trail is the relevant evidence. What the team is actually deciding is the relevance argument: which features enter the system, and what the app will claim when they point the wrong way. That decision arrives through design choices rather than through the definition. Dey, Abowd and Salber (2001) bounded the Context Toolkit's scope to context inferable from sensors in a physical environment, which excludes whatever the user would have to state aloud, so the toolkit both hides the argument and narrows it. Schilit, Adams and Want (1994) had already shown that context can be active, supplied by the user, so the team could make part of the decision explicit by offering a stated signal rather than guessing. The team should also expect the situation to be produced in the moment rather than fixed in advance: Dourish (2004) argues context is an interactional achievement, and Bellotti and Edwards (2001) argue that some context cannot be sensed or even inferred, so a system that acts unasked must be able to defer and to explain itself. Guide the team to check that against the field record: Cheverst and colleagues (2000) found that the visitor their GUIDE system modelled was not the visitor whose behaviour the trial observed, and Chen and Kotz (2000) found that such applications had never become widely available to everyday users. The relevance argument is also a bet about transfer, and Meegahapola and colleagues (2024) show both that the bet fails when the deployment distribution differs from training, and that unsupervised adaptation across 14 source-target domain pairs can recover up to 12 per cent in area under the curve against direct deployment. A test that would settle whether the design works is not a satisfaction survey but a comparison holding the sensed features fixed while varying whether the app defers; if the interactional position is right, the deferring version should score better on intelligibility and on outcomes, especially where users know more about the situation than the sensors do.

The number is an estimate, not a measurement, and this is the first thing the team has to accept. Kalman (1960) showed that a filter updates a state estimate together with the covariance of the estimation error, and that its optimality is conditional on a linear system with Gaussian noise; displaying one decimal place asserts a precision that the covariance contradicts and that the model assumptions may not support. The next question is what governs the error, and the accelerometry literature answers it before the team writes any code. Yang and Hsu (2010) report that the accuracy of energy-expenditure estimation depends on the location and attachment of the accelerometer, on external vibration and gravitational artefact, and on the activities performed in a free-living environment, and that loose attachment degrades the signal; the waist placement they describe is chosen because it lies near the centre of mass, not because it is neutral. A phone in a pocket has no fixed attachment and no fixed orientation, so the mounting is a source of systematic error the team does not control. Mathie and colleagues (2004) show the positive version of the same finding, that a single waist-mounted unit can serve several parameters under unsupervised monitoring, which means the sensor is capable once its conditions are stated. If the team adds a heart-rate sensor to repair the accelerometer's weakness, Gravina and colleagues (2017) supply the vocabulary for what that decision is, fusion at the data, feature or decision level affecting application performance, and Khaleghi and colleagues (2013) supply the warning, that measurements always carry imprecision and uncertainty and that a fusion algorithm must express these rather than assume them away; Aziz and Koo (2025) state the same trade for hybrid location, where combining technologies mitigates drawbacks rather than eliminating them. The evidence that would settle whether the product is good enough is a factorial comparison rather than a single accuracy figure: the same activities and the same population, with mounting, activity set and any additional sensor varied independently, reporting the distribution of error and labelling where the estimate fails, since a mean error hides the tail that the product must survive. The team should also read Harle (2013), whose finding that a sensor can be reliable over one timescale and not another is the general form of the warning.

The first thing to say is that the figure establishes performance under the conditions the benchmark fixed, and nothing more. Anguita and colleagues (2013) show what a benchmark fixes: thirty volunteers, six activities and a waist-mounted phone. Every score computed on that partition inherits those choices, so 94 per cent is a statement about that population, that placement and that activity set. Bao and Intille (2004) show how much those choices matter, since some activities were recognised well from other people's data while others required the wearer's own data, and cutting to two sensors cost little. The team should therefore expect the number to move as soon as the population or the placement changes. The device axis is the second problem. Stisen and colleagues (2015) tested thirty-six smartphones and smartwatches across thirteen models from four manufacturers and found that sensor and sensor-handling differences impaired performance significantly, with deployed performance often lower than the reported research result. A recogniser shipped to whatever phones users own will meet exactly that heterogeneity, so the benchmark score is not a deployment estimate. The team should also check the protocol behind the number. Reyes-Ortiz and colleagues (2016) show that segmentation and the treatment of transitions change the measured result, and Lara and Labrador (2013) report that within-dataset evaluation is the norm and that this practice overstates performance. The evidence that would show generalisation is a held-out evaluation: train on one group of people and one set of devices, test on people and devices the model never saw, and report the result per participant and per device rather than as one pooled figure. Mitchell and colleagues (2026) move in this direction by releasing real-world data labelled by activity and terrain, and Yin and colleagues (2024) confirm that most studies still evaluate within self-collected datasets, so this test is still rare. If the team adds a deep model to raise the score, Hammerla and colleagues (2016) and Ordonez and Roggen (2016) show that the gain is real but measured within the training conditions, and Wang, Chen and colleagues (2017) note that no single model wins everywhere and that transfer remains an open challenge. The recommendation is to report the score with its population, its devices, its activity set and its segmentation, and to treat the benchmark figure as a ceiling under chosen conditions rather than as a promise about strangers.

The first point is that a seated desk test samples one posture out of the set the product will meet. Azenkot and Zhai (2012) collected text entry data from 32 participants in three postures, two thumbs, one thumb and one index finger, and found accuracy and precision differed systematically between them, with consistent offsets an algorithm could correct. A layout that works with two thumbs on a desk is therefore not evidence that it works one-handed. The second point is that the top corner is likely outside the thumb's reach. Parhi and colleagues (2006) found that target position changed performance and recommended 9.2 mm targets for discrete tasks and 9.6 mm for serial ones while varying screen position, and Bergstrom-Lehtovirta and Oulasvirta (2014) showed the thumb's functional area can be computed from surface size, hand size and index finger position, so the reachable region is a measurable constraint rather than a matter of taste. Wobbrock, Myers and Aung (2008) add that posture and movement matter as much as the hand, with two-handed postures faster than one-handed ones and the index finger faster than the thumb. The third point is movement. Goel, Findlater and Wobbrock (2012) showed that reading the accelerometer and correcting keystroke interpretation cut uncorrected errors by 45.2 per cent and raised speed by 12.9 per cent for walking participants, so the team should treat walking as a condition to design and test for, not as an occasional nuisance. The fourth point is use pattern. Bohmer and colleagues (2011) logged usage from over 4,100 users and found the average session lasted under a minute although almost an hour a day went on the phone, and Oulasvirta and colleagues (2011) describe checking as a brief, repeated habit reinforced by immediate rewards, so reach and latency matter more than throughput. On testing, the team should place controls inside the measured thumb area, test with the postures and the walking condition the product will actually meet, and report results per posture and per participant rather than as one pooled figure, since Dhakal and colleagues (2018) separated most typists into eight groups and Kane, Wobbrock and Ladner (2011) found blind users' gestures differed from sighted users' in speed, size and shape. If the team adds sensing to adapt the interface, they should note that swipe trajectories carry demographic and behavioural correlates (Lemarquis and colleagues, 2023), so the adaptation channel is also a profiling channel.

The first point is that a laboratory day measures one device, one workload and one place, none of which the shipped product can assume. Falaki and colleagues (2010) found diversity of an order of magnitude or more across 255 users, with between 10 and 200 interactions a day, so the team's test workload is one sample from a wide distribution. Chen and colleagues (2015) found across 1,520 smartphones in the wild that the cost of the same workload varies with hardware, operating system and application updates, so a result on one prototype phone does not transfer. The second point is that the upload schedule, not the data volume, governs the radio cost. Balasubramanian, Balasubramanian and Venkataramani (2009) found that nearly sixty per cent of a typical 3G transfer is tail energy spent after the bytes move, that a few hundred bytes sent intermittently can cost more than a megabyte in one transfer, and that deferring and batching cut modelled energy by 35 to 52 per cent; Huang and colleagues (2012) found the same tail pattern on LTE, where a single packet cost 12.76 joules against 0.04 on WiFi. The team should batch the summaries and upload them together rather than several times an hour on a fixed timer. The third point is that continuous sensing changes which costs dominate, because the device must act even when nothing is happening. Priyantha, Lymberopoulos and Liu (2011) built LittleRock to keep the application processor asleep while a low-power microcontroller handled the sensor stream, which shows the sampling architecture is a scheduling decision, and Pathak and colleagues (2011) show that sensing loops repeatedly wake drivers and radios whose wake costs are fixed and repeated, below the application where the developer does not see them. The fourth point is that the environment is not under the team's control: Ding and colleagues (2013) found across 3,785 smartphones that weak signal is routinely experienced and raises the transmit power for the same data, so the same upload costs more in poor coverage. On measurement, the team should use tools that show energy below their own code, such as the system and per-application estimation of Zhang and colleagues (2010) and Yoon and colleagues (2012), and should treat code-level choices as energy decisions, since Groza and colleagues (2024) found correcting energy code smells cut drain by 4 to 30 per cent and Caiazza and colleagues (2024) found a protocol version changed the energy of the same request by roughly 9 to 12 per cent. They should report the budget per device and per signal condition, not as a single laboratory figure, and present it to the user as a prediction under a stated pattern of use, following Oliver and Keshav (2011).

The evidence supports the mechanism. Iqbal and Bailey (2008) found that deferring notifications to detected breakpoints reduced frustration and reaction time against immediate delivery, at a cost of about a minute and a half, although their models detected breakpoints reasonably well while struggling to differentiate their type; Fischer, Greenhalgh and Benford (2011) found that notifications delivered after an episode of mobile interaction were attended to significantly more quickly than against a random baseline; and Okoshi and colleagues (2015) found 46 per cent lower cognitive load at detected breakpoints than at random timing, and a 33 per cent decrease in a 16-day in-the-wild study with 30 participants. Horvitz and Apacible (2003) supply the framing: the cost of interrupting is an estimate under uncertainty, which is what the system is computing. The evidence does not support the claim that content can be ignored. Iqbal and Bailey's own finding is that content relevance determines which breakpoint suits a message. Mehrotra and colleagues (2016) found that even a notification with important or useful content caused disruption and that presentation, alert type, sender-recipient relationship, task complexity and individual psychological traits all moved the outcome. The feature will serve worst the users whose reason for checking the phone is social: Pielot, Church and de Oliveira (2014) found 63.5 notifications a day typically viewed within minutes whether or not the phone was silent, with social pressure among the main reasons. Pielot and Rello (2017) found that 30 volunteers who disabled alerts for 24 hours were less distracted and more productive but also anxious about being less responsive and less connected. The team should therefore treat deferral as a social decision, hold personal messages briefly and visibly rather than silently, offer per-application and per-keyword control of the kind Kim and Park (2025) evaluated, and report what a deferred message cannot do: recover the social cost. Measurement should follow Mark, Gudith and Klocke (2008), who found interrupted work finished sooner without loss of quality but with more stress, frustration, time pressure and effort, so completion time alone will understate the cost; and Mark, Gonzalez and Harris (2005), who found 57 per cent of working spheres interrupted with more than two other activities intervening before resumption. The team should also answer the survey's criticism, reported by Mehrotra and Musolesi (2017), that studies use small samples for short periods and are validated offline, by reporting a field trial with a stated sample, its duration, and the share of messages whose deferral exceeded the two to five minutes that inattentive periods typically last.

The permission model does not do the work the team claims. Acquisti, Brandimarte and Loewenstein (2015) found that stated concern about privacy often diverges from actual disclosure, driven by context and framing, so a prompt at install is exactly the frame that fails to predict behaviour; Solove (2007) adds the conceptual reason, that privacy is not secrecy but protection against aggregation, inference and use, so consent to one disclosure does not address the harm of the collection. The history is the exposure. Beresford and Stajano (2003) argued that a fine-grained trace is a different kind of object from a single fix, and Krumm (2007) showed that a user's home can be inferred from sparse, noisy traces, so the rolling summary reveals more than the positions it appears to show. Re-identification is measured, not feared: de Montjoye and colleagues (2013) found four spatiotemporal points uniquely identified 95 per cent of people, robust to coarsening, and de Montjoye and colleagues (2015) re-identified 90 per cent from four credit card transactions without location at all; Zang and Bolot (2011) showed anonymisation fails on data already published. The team should treat local processing and deletion as design decisions, not settings. Before release it needs a real evaluation against an explicit adversary of the kind Shokri and colleagues (2011) formalise, with re-identification and utility measured on the same population; a formal guarantee rather than an informal promise, in the sense of geo-indistinguishability (Andres et al., 2012); attention to inference beyond location, since Qiu and colleagues (2025) found that patterns of service use reveal sensitive attributes; and, where any aggregate is published, a synthesis method such as LDPTrace (Du et al., 2023) rather than a release of raw traces.

The team's premise is the one the field studies contradict. Brush and colleagues (2011) visited fourteen households that already had home automation and found the sensing worked while adoption had not followed, with four barriers, cost of ownership, inflexibility, poor manageability and difficulty achieving security, all of which concern installation and upkeep. Edwards and Grinter (2001) had already argued that design for the home must rest on how domestic technology is actually adopted and used, and Tolmie and colleagues (2002) supply the reason the team's model matters less than it thinks: a system that must be commissioned, named and repaired keeps returning as a task, so it cannot become unremarkable. Jakobi and colleagues (2017), in an eighteen-month living lab study of fourteen households, found the difficulties concentrated at setup and configuration, and Hargreaves and colleagues (2017), following ten households over nine months, found the technologies disruptive, demanding of adaptation and familiarisation, and requiring learning that was time-consuming with little support, with little evidence of substantial energy savings and a risk of energy intensification. That last finding bears directly on an energy product, because the promised saving may not appear. Davidoff and colleagues (2006) found families wanted control of their lives rather than of their devices, so the team should design for the household's authority over the system, including the ability to see and correct its decisions, as Yang and Newman (2013) argue from twenty-three thermostat interviews in which users built workarounds because they could not understand the system's behaviour. Wilson and colleagues (2017) add that the perceived risk is ceding autonomy and independence, and that marketing in this sector under-emphasises data security and privacy, so the security posture must be visible rather than assumed, and Peterson and Mujeye (2025) show how little households do in practice, with only 24 per cent updating software regularly and 29 per cent having a plan for vulnerabilities. The team should measure adoption, abandonment and the time the household spends on installation, naming, correction and updating, over months rather than weeks, and report savings from metering. The disagreement to settle is whether the barrier is household work or, as Wu, Yang and Sun (2026) argue, the data and generalisation limits on deployment; the team can contribute the missing comparison only if it holds the model fixed and varies what the household must do.

The trust should read the laboratory validation as evidence about a chosen set of conditions and no more. Duh, Tan and Chen (2006) evaluated one mobile device in both a laboratory and real-life situations and found significant differences in the frequency and severity of problems, in behaviour and in subjective responses, so a laboratory result is not a cheaper version of the same finding. The trust should therefore ask for the reliability evidence, not only the accuracy evidence, because Cvach (2012), reviewing 72 articles, ties alarm fatigue to a high false alarm rate and poor positive predictive value, and Bliss, Gilson and Deaton (1995) showed with 138 undergraduates that about 90 per cent match their response rate to the expected probability that an alarm is true. Configuring thresholds and delays is a reliability decision, and it is the one the supplier has left to the purchaser. Edworthy and Hellier (2006) found alarms too loud, too numerous and consequently turned off, which is what a locally tuned monitor will become if the trust sets levels by preference rather than by measurement. The trust should therefore measure the alarm stream before and after configuration and report composition as well as volume, as Lewandowska and colleagues (2026) did in one unit, where volume rose by 27.4 per cent while red technical alarms fell by 61.1 per cent and staff still reported fatigue. Finally, the trust should recognise that the laboratory result may not travel: Wickens and colleagues (2009) analysed 495 real conflict alerts and found no failure or delay in responding to true alerts, because controllers anticipated them. Only local data linking false-alarm rate to responses to true events, collected over months, can settle that for this trust, and the local configuration is the experiment that produces it.

Use Reason's person and system distinction as the frame. The unit entry is an active failure at the sharp end, and the first move is to say what kind of act it is rather than who performed it. Nothing in the account suggests a deliberate breach, so classifying it as forgetfulness and stopping there would leave the conditions untouched. The systemic search then runs upstream. Was the unit mismatch an identified use error in the pump's hazards? What are the device's default settings and does the interface display the unit next to the field? Where does the rate get set, in what conditions, and with what interruptions? Has the same pump produced near misses that were never recorded? Safety-II adds a second question: how does the ward usually get this right, and what does that workaround look like on a busy night? Two pieces of evidence would come first. One is the pump's event log and configuration defaults, read alongside the manufacturer's usability file, to establish whether the unit ambiguity was a known hazard. The other is the ward's incident and near-miss reports plus a short observational study of how rates are set, to see whether this act is a pattern rather than an outlier. The recommendation should change the conditions, for example unit-consistent fields and a hard confirmation of unusual rates, not the person alone. Commit in advance to what would show it worked: a measured fall in wrong-rate events and near misses, not a completed retraining record.

Separate the two claims, because they need different evidence. The reduction is a before-and-after comparison with no control group, so it cannot attribute the change to the app. Other service changes, seasonal variation, a change in who was referred, or better recording would all produce the same figure. You also need the denominator and the uncertainty around the estimate before treating 12 per cent as a result, and you need to know whether the outcome was specified before the data were analysed. The safety claim is a different kind of claim and the weakest part of the report. Safety is not established by an outcome study or by a usability test. If the app is a medical device, the manufacturer must run a usability engineering process under IEC 62366-1, which requires the risks of correct use and of use error to be assessed and mitigated, and the standard states that it cannot assess the risks of abnormal use. If it is not a device, ask what hazard analysis was done, which harms were looked for, and how they were monitored after release. Blandford and colleagues' distinction is useful here: formative evaluation asks how a design performs with users and feeds iteration, summative evaluation asks whether the intervention worked against a comparison, and the report offers neither convincingly. Require a controlled design with a pre-specified primary outcome for the benefit claim, and the hazard analysis, use specification and use-error evidence for the safety claim.

The draft report is the person approach in its familiar form, and Reason (2000) supplies the terms for saying so: it reads a sharp-end unsafe act as a product of inattention and reaches for retraining and another procedure, the remedies he records as having only very limited effectiveness, especially with trained professionals. The unit's alternative is to treat the act as a symptom and ask what in the system made it likely, and the record already contains the candidate: a nurse covering two bays. Reason (1995) gives the team a place to put it, since safety-significant errors occur at all levels and latent failures in organisational and managerial spheres are present long before an accident. The taxonomy can support the classification, and that is all it can do: classifying the omission as a lapse tells the team it was an execution failure rather than a failure of planning or problem solving, which changes the countermeasure from retraining in a skill to a check on task load. Rasmussen (1982) sets the limit, warning that analysts tend to accept human error as the explanation by the causal backtracking "it is human to err", so the report has to look for external causes as well. Woods and colleagues (2010) set a second limit: what is labelled error is a judgement made after the fact, so the team should record the judgement and the standpoint it comes from. Shappell and colleagues (2007) show that trained raters can apply such a taxonomy consistently, and Ahmadi Rad and colleagues (2023) record that the layered-defences model has been criticised for treating contributing factors as conditions rather than as interactions, which is the risk in a report of this kind. What would settle the question is an outcome test: a setting where a change in classification changed the countermeasure and the harm rate, which the unit's sources do not provide.

The proposal begins from the right observation and draws the wrong inference. Hollnagel, Wears and Braithwaite (2015) define Safety-II as the system's ability to succeed under varying conditions, and the board's premise, that staff already hold work together, is consistent with that. But the instrument it reaches for measures something else. Woods and Hollnagel (2017) place resilience in the system through four abilities: to respond, to monitor, to anticipate and to learn. Thielmann and colleagues (2024) surveyed 285 emergency medical services personnel with the RS-13 resilience scale and found 39 per cent with a low level of resilience, in a cross-sectional design with no measure of system performance. A before-and-after staff score therefore cannot show that the ability of the ward to succeed has changed. The board is also committed to a position: it treats safety as a set of properties of individuals, which is the drift Woods (2015) warns about when he separates rebound, robustness, graceful extensibility and sustained adaptability and argues that the field must choose one meaning before resilience can be measured. A Safety-II programme would ask how medication work usually goes right, which is the move Sujan, Huang and Braithwaite (2017) propose when they argue that organisations should learn from everyday clinical work as well as from incidents, and it would follow the worked approach the CARE programme takes, using the Concepts for Applying Resilience Engineering model to study adaptations and then to design interventions with clinical staff (Anderson et al., 2016). Fairbanks and colleagues (2014) introduced the vocabulary to clinical quality work in a commentary that summarises a workshop and gives worked examples of resilience engineering techniques rather than an evaluation, which is the evidence base a board should notice before adopting a programme. What would settle whether the proposal works is an outcome rather than a score: the deferred evaluation phase of a programme such as CARE, a resilience measure with demonstrated predictive validity, or a comparison of the countermeasures that Safety-I and Safety-II accounts of the same incidents generate, measured on harm.

The proposal confuses a measure of awareness with evidence of benefit, and it chooses an instrument the unit's sources show to be unvalidated. Endsley (1995a) defines situation awareness as a three-level construct separated from decision making and performance, so a change in an awareness score is not itself a change in patient outcome. The questionnaire is the harder problem. Ghaderi and colleagues (2023) reviewed situation awareness instruments for health care providers using COSMIN, including twenty-five studies and fifteen instruments, and found that no instrument reported all measurement properties and that evidence quality was not high in any; cross-cultural validity and responsiveness were not investigated at all. A before-and-after score from one of those fifteen instruments therefore cannot show that the dashboard improved safety, because the instrument is not yet established and because responsiveness, the very property needed to detect change, has not been tested. By measuring each nurse separately the team also commits to the individual account of awareness that Stanton and colleagues (2006) and Salmon and colleagues (2008) argue is the wrong unit for collaborative work: on the distributed account, each agent holds a compatible but different picture, so a distributed study would examine the interaction between nurses, the dashboard and other staff, following Gorman and colleagues (2006) by looking at what the team does rather than at what each member scores. The proposal also inherits the automation dispute. If the dashboard is automating detection, Endsley and Kiris (1995) show that awareness and takeover can fall as operators shift from active to passive processing, while Norman (1990) argues the harm lies in feedback and interaction design rather than in automation as such. What would settle whether the dashboard improves safety is a comparison against the same task with different feedback, plus an outcome measure such as time to escalation or harm, ideally with an instrument that has demonstrated predictive validity; a before-and-after questionnaire score alone meets none of those conditions.

The proposal uses a real law to answer a question it was not built to answer. Dawson and Reid (1997) measured hand-eye coordination on an unpredictable tracking task in 40 subjects, in two counterbalanced conditions, and reported that performance fell by 0.74 per cent per hour between the tenth and twenty-sixth hours of wakefulness, with 17 hours awake equivalent to 0.05 per cent blood alcohol and 24 hours to about 0.10 per cent. That is a dose-response relation for one outcome measure in a laboratory, so it supports a claim about sustained wakefulness and that task. It does not establish what a reduction in shift length does to a ward, because the effect size is conditional on the task: See and colleagues (1995) pooled 42 studies and 138 conditions and found the sensitivity decrement depended on the discrimination type and the event rate, so the same exposure yields different costs in different task settings. The sleepiness rating is the weaker half of the proposal. Ratings are informative within a person over time, but Van Dongen and colleagues (2003) found that sleepiness ratings did not significantly separate the 6-hour from the 4-hour condition while performance deficits accumulated dose-dependently, and Akerstedt and Gillberg (1990) found that physiological signs of sleepiness appeared reliably only once self-rated sleepiness was already high on the nine-point scale. The measure therefore saturates in the range a night shift occupies, and it cannot detect the deficit it is being used to rule out. A study that could test the transfer would keep the law and add what the ward supplies. It would record a performance measure on the work itself, not only a rating; it would measure the performance obstacles that Gurses, Carayon and Wall (2009) found to raise intensive-care workload and to mediate perceived quality and safety of care; and it would test whether shortening the shift changes those obstacles or only the nurses' fatigue. Hockey's (1997) compensatory control framework supplies the reason such a study cannot rely on outcomes alone: primary-task performance can be protected by recruited effort, with the cost appearing in subsidiary tasks, strategy and strain. Finally, the proposal commits the trust to the resource account of the decrement, since it treats the deficit as a dose of exposure that shorter shifts will reduce, and that account is contested by Thomson and colleagues (2015) and by Skinner and colleagues (2023).

Programme A commits the unit to the exposure position, that the number of alarms does the damage. That position has real support: Bonafide and colleagues (2015) found response time to alarms raised while no clinician was in the room rising with the number of non-actionable alarms in the preceding 120 minutes, and Dursun Ergezen and Kol (2020) found response rates falling as the false alarm rate rose. The measure has two weaknesses. First, the justification for the target is the very quantity under test, since Sowan and colleagues (2016) cut the alarm rate from 87.86 to 59.18 alarms per patient day and found nurses' attitudes unchanged, with half the nurses wanting more training. Second, the count says nothing about whether the alarms that remain are the ones that matter, and Rayo and Moffatt-Bruce (2015) argue explicitly that total alarm rate is not supported as a driver of response. Programme B commits the unit to the informativeness position, which is better grounded in the response literature and comes with a stated procedure for measurement, but informativeness runs between 1 and 20 per cent across healthcare, which means a 40 per cent target is unprecedented and unvalidated. Its survey is the weakest part, because the construct being surveyed has no agreed definition and was not measured directly in the intervention literature reviewed by Winters and colleagues (2018), although a validated questionnaire has since been published (Wunderlich et al., 2024). An evaluation that could test either programme would need a direct alarm-fatigue metric, response time to actionable alarms, and a count of actionable alarms missed, since Gazarian (2014) shows that a low overall response rate, 46.8 per cent, can coexist with no adverse event. Without the missed-actionable outcome the unit cannot show that fewer alarms did not mean fewer answers.

Neither measure engages the mechanism the unit identifies. The first failure is Ash, Berg and Coiera's (2004) communication and coordination family: the record lets a clinician treat entering an order as the whole communication of the plan, so the order is never spoken about and the omission leaves no trace to be found later. A newer system buys a different set of fields, not a different relationship between recording and telling, and Ash and colleagues (2007) found that the kinds of unintended consequence reported by 176 hospitals bore no relation to how long the system had been in use, so the promise that a newer product fixes an old one is unsupported. Training assumes the hazard lies in the user, and Baysari and colleagues (2011) show it does not: senior doctors made the prescribing decisions while the junior doctors who entered the orders received the alerts and ignored them, so the failure is a targeting failure in the design, not a knowledge gap. Ratwani and colleagues (2018) found nine-fold differences in time and error rates up to 50 per cent between sites running the same software, which makes implementation rather than purchase the variable to work on. An evaluation would need to measure whether orders were acknowledged by the clinician expected to act, how often alerts reached the decision-maker, and the rate of actionable orders missed, because Howe and colleagues (2018) show that reported harm events are a floor and cannot carry the verdict alone.

A two-ward analysis can support a diagnosis and cannot support the board's expectation. Meilan and colleagues (2025) show precisely what a well-executed SEIPS 101 analysis yields: six phases of counting in four environments, eight people influencing the process, and 56 barriers against 31 facilitators, all from 22 observed cases. That is a structured account of where the work fails, produced before any intervention, so it establishes nothing about improvement, and it belongs to one process in one hospital. Xie and Carayon (2014) reviewed 12 projects representing 23 studies of human factors based redesign, found some evidence of effectiveness with varying study quality, and asked future work to define the mechanisms of improvement, which means the trust's expectation of transferable error reduction is not something the literature currently licenses. Carayon and colleagues (2014) add the design constraint: because balancing the work system is the principle, a remedy aimed at one component can shift load onto another. The contested question bears directly on the decision. Waterson and colleagues (2015) found that current methods capture influences external to the studied system poorly, so the organisation and environment components carrying most explanatory weight may fall outside the analysis, and Rasmussen (1997) locates the levers that produced the conditions for error above the ward. Salmon and colleagues (2022) show the propositions themselves still being negotiated. What would change the position is a study design: specify the analysis and the predicted effect before the intervention, repeat it across wards, and measure an outcome that could falsify the claim. Holden and Carayon (2021) published the tools that make such comparison possible, so the means exist; a two-ward diagnosis alone is not the evidence the board needs.

The file supports a narrow and useful claim. The manufacturer has followed the process IEC 62366-1 (2020) specifies: it has analysed, specified and evaluated usability as it relates to safety, and it has tested the critical tasks with participants that satisfy the FDA guidance (2016), which asks for at least 15 participants per distinct user population and treats nurses and lay caregivers as distinct populations. That is a competent evidence base, and it is more than many submissions carry, since Vincent and colleagues (2014) found that communication barriers between development teams and users are a routine constraint on designing for safety. What the file does not support is any claim about use error that the protocol did not define as failure. Furniss and colleagues (2011) recorded 10 low-level disturbances across 31 observations of pump programming in practice, and those disturbances completed the task; a validation report in which no use error was observed says nothing about them. Nor does the file state what the evidence is meant to establish. Bishop and Bloomfield (1998) require a safety case to hold claims, evidence, the argument linking them and the inference rules, so the missing artefact is the argument rather than the evidence. Two things are left to the manufacturer's judgement. IEC 62366-1 can identify but not assess or mitigate the risks of abnormal use, so that class falls outside the obligation, and ISO 14971 (2019) requires objective criteria for risk acceptability without specifying acceptable risk levels. The manufacturer therefore sets its own threshold for the residual risk it will accept. That is the disagreement the unit sets out. The first position treats the obligation as adequate because it is countable and external, and reads the thinness of practice as an enforcement problem. The second holds that a countable obligation is not a stated claim. What would settle it is a comparison, not another position statement: take device families of comparable risk, one with an explicit claim-evidence argument that names its residual risks and one with a validation report alone, and compare post-market use-error reports. The falsifiable version should be the one unsurprised by the errors it later records.

The proposal treats volume as a proxy for safety, and the literature rejects that reading. Vincent, Stanhope and Crowley-Murphy (1999) found that readiness to report a designated incident varied from 96 per cent for a maternal death to fewer than 40 per cent for a baby's unexpected admission to the special care baby unit, and that almost 30 per cent of staff could not find the list of reportable incidents; their conclusion is that such systems do not give a reliable index of the rate of adverse incidents. Macrae (2015) adds that a focus on quantity over quality ignores learning, and that when call volumes are used for summative judgement the incentive is to game them. Kaya and colleagues (2023) show reporting volume tracks culture, but their models predicted it without high accuracy, so a monthly count is a weak signal about compassion, work pressure and harassment rather than a measure of safety. Landrigan and colleagues (2010) supply the outcome test: ten North Carolina hospitals reviewed from 2002 to 2007 showed a rate of 25.1 harms per 100 admissions and no significant change in harms per 1,000 patient days, so a decade of activity did not move the rate. The previous investigations are explained by the investigation literature rather than by staff effort. Nicolini, Waring and Mengis (2011) found that root cause analysis is used as a governance tool and legitimacy device as well as a learning technique, and that these conflicting agendas divert effort from sustainable change. Peerally and colleagues (2016) argue the method has been applied without attention to what makes it work in its original contexts or adaptation for healthcare, and that organisational forgetting remains widespread; recommendations for refresher training are exactly the under-specified output they criticise. Kirkup (2015) shows the consequence at scale: in maternity services at Morecambe Bay from 2004 to 2013 the response to adverse incidents was grossly deficient, with repeated failure to investigate properly and learn lessons throughout the period. A just culture, on Dekker's account, is not achieved by publishing numbers; all current proposals assume culpability inheres in the act, when the critical question is who draws the line between blameless and blameworthy conduct, and Dekker and Breakey (2016) add that a genuinely just culture needs legitimate rules, fair procedures that protect the accused, and restoration of those affected. Publishing a monthly count without any of that risks deterring reports from the staff who currently under-report, which is what Vincent and colleagues documented. What would show the proposal working is not a larger number but a documented change in practice following a documented change in the system, measured against an outcome such as the rate of delayed recognition, using a design that repeats a blinded review of cases before and after the change as Landrigan and colleagues did. The eight challenges set out by Peerally and colleagues give the intermediate test: whether the next investigation names a systemic condition and produces a recommendation that is implemented and audited. On the evidence of this unit, doubling the reports without changing what investigations are for is more likely to add to the archive than to reduce harm.

The two documents answer different questions, and the director's error is to read them as one proof. Blandford and colleagues (2018) set out the contrast directly: formative evaluation, emphasised in Human-Computer Interaction, and summative evaluation, emphasised in health research, differ not only in timing but in the nature of accepted evidence and the culture of reporting, which is why they write seven lessons rather than a shared protocol. A usability report is evidence about whether people can operate the product; it is not evidence about whether they will keep using it or whether their condition improves. The trial has the reverse limitation. Murray and colleagues (2016) argue that a trial is best undertaken only when the intervention and its delivery package are stable, can be implemented with high fidelity, and there is a plausible reason to expect benefit, so the trial result is a claim about the product as it was tested rather than about the product as it will be deployed. The real-world engagement evidence shows how far apart those can be. Baumel and colleagues (2019) found a median daily active user rate of 4.0 per cent and median retention of 3.9 per cent at 15 days and 3.3 per cent at 30 days across 93 Android mental health apps with ten thousand or more installs, and Fleming and colleagues (2018) found completion or sustained use ranging from 0.5 per cent to 28.6 per cent for implemented self-help interventions. If effect depends on use, a trial that recruited volunteers under research conditions measures an effect that the service's population may not reproduce. Engagement itself is contested as a construct: Perski and colleagues (2016) show it conceptualised in both experiential and behavioural terms, and Yardley and colleagues (2016) argue that effective engagement with the target behaviour matters more than more engagement, so the service should specify which form of use it needs the clinical benefit to come from. Adoption is a separate evaluation with its own dimensions. Greenhalgh and colleagues (2017) report that programmes with complexity in multiple domains rarely became mainstream, and Marwaha and colleagues (2022) set out nine dimensions a health system should examine before adopting a validated tool, noting that few strategies exist to help tools cross the chasm from clinical validation to integration in large systems' workflows. The director should also note what the evidence archive typically omits: Enam and colleagues (2018) found that evaluation is performed at the end of an intervention and that case studies assessed only the clinical and the human and social aspects, so implementation and cost questions are usually unexamined. Finally, the framework that governs the purchase is conditional rather than general. NICE (2022) assigns a technology to a tier and scales the evidence requirement to that tier, and states that meeting the standard is neither endorsement nor regulatory approval, so the service must state which claim it is making at which tier. What would settle the question is a prospective study of this product across its lifecycle: objective and pre-specified engagement outcomes of the kind Baumel and colleagues measured, a summative trial meeting the conditions Murray and colleagues set, the product's tier and therefore its evidence requirement declared in advance, and the formative work reported beside the summative result. No source in this unit reports such a study, and on Guo and colleagues' (2020) account the evaluative methods have not kept pace with the products, which is precisely why the director's two documents cannot be added together into a decision.

A strong answer holds the three categories apart rather than treating them as degrees of pressure. Persuasion gives reasons the user can accept, such as a genuine discount or faster delivery. Manipulation subverts the user's reasoning, for example by fabricating scarcity so that a rational assessment cannot be made. Coercion removes the choice, which is not in play here. The first question is therefore factual, not moral: is the scarcity real? If stock truly is low and the timer tracks a real deadline, the pattern is persuasion, and the commercial case is legitimate. If the message is generated per visitor or the timer resets on reload, the design is a deliberate attempt to defeat considered choice, and no amount of disclosure repairs that. The legal instruments follow from the facts. Under the UK GDPR, a claim that misleads a person into a purchase may undermine the fairness of the processing that supports it. Regulation (EU) 2024/1689 prohibits artificial intelligence systems that deploy manipulative or deceptive techniques, which matters if the message is generated by a model rather than by a rule. The evidence needed is specific: the stock query that populates the message, a log showing whether the timer is per-session or per-visitor, and a test of whether the same item shows a different figure to a different account. Without that evidence the argument is an appeal to taste, and it will lose to the commercial case. With it, the question becomes whether the team meant to deceive.

A defensible audit separates what can be counted from what must be judged. The countable part follows Mathur and colleagues: define the pattern types in advance, script an automated crawl of the pages that matter, and record every instance with a screenshot and the DOM evidence so that another person can reproduce the finding. Report the count against the number of pages examined, because a bare total says nothing without the denominator. For fairness, define the groups and the decision being tested before looking at the results, and prefer a measured outcome such as the error rate for each group over a claim about intent. Then state the limits plainly rather than burying them. The crawl observed one version of each page, so personalised, segmented or experimentally varied designs are invisible to it, which makes the total a floor and not a prevalence estimate. The detection rules encode the auditor's assumptions, so a pattern outside those categories is not detected rather than absent. A single observation cannot show that a pattern changed a decision, only that it was presented. And unequal outcomes across groups do not identify a cause, so the audit can raise a question but cannot settle it. That is the argument Raji and colleagues make for moving the work inside the organisation and running it across the whole life cycle, where the design decisions, the data and the model versions are documented as they happen. An external audit is a photograph of a moving system; an internal one is a record of how it was built.

Separate the record's claims. First, the convergence is real but proves less than the team thinks. Moor (1985) explains why the frameworks were never interchangeable: computer ethics answers questions no earlier framework anticipated, so agreement on a verdict is not agreement on a reason. The consequentialist verdict rests on outcomes, the deontological verdict on a right, the virtue verdict on the disposition behind the act, and each addresses a different audience, as Bynum (2006) and Vallor (2016) show for the virtue arm. The team should say which reason it relied on, because that is the reason it must defend. Second, the claim that value sensitive design guarantees that values reached the specification overstates the method. Friedman (1996) genuinely moved ethics from verdict to property by treating a value as something a design can be specified to support, and van Wynsberghe (2012) shows the method producing design requirements rather than a restatement of care ethics. But Le Dantec, Poole and Wyche (2009) argue its mechanics create tensions with value sensitivity, and Borning and Muller (2012) say it overclaims and needs contextualised value lists. Third, Manders-Huits (2010) supplies the decisive objection: the method has no explicit ethical theory for value trade-offs and leaves the concept of values undetermined, so it names the values without deciding between them. The team has shown vocabulary, not a decision rule. To show its values actually reached the specification the team would need what the unit says the literature has not produced: a comparison of decisions made through the method with decisions made through one framework alone, on whether they are more defensible or produce better outcomes. Portions of this answer are synthesised from the cited sources rather than quoted from any single one.

Separate the record's claims. First, the codes bind a person and not the firm. The ACM code was adopted on 22 June 2018 and binds members, SIG members and award recipients (ACM 2018), while the BCS code of conduct, version 8, states that it governs the individual and not the ethics of a relevant authority, which it defines as the employer or client (BCS 2022). Following the code therefore satisfies the member's duty and places no obligation on the team to change the product. Second, the claim that the code covers the ethical questions overstates the evidence. McNamara, Smith and Murphy-Hill (2018) found that instructing 63 students and 105 professional developers to consider the ACM code had no observed effect on their responses to 11 ethical vignettes, so reading the code is not what changes a decision. Third, the logged concern is exactly the case the literature prices. Mesmer-Magnus and Viswesvaran (2005) found that organisational context predicts both whether a concern is raised and what follows for the person who raised it, and Bjorkelo (2013) links whistleblowing to subsequent workplace bullying and its effects on health. The engineer has absorbed a cost and changed nothing. The unit's organisational evidence says where the repair belongs: Madaio and colleagues (2020) show that checklists can supply organisational infrastructure and empower individual advocates, and Vakkuri and colleagues (2020) name documentation and error handling as existing practices that can carry ethics. To make the concern change the product the team needs a retention decision owned by someone with authority, a documented retention period tied to the purpose, and a route that does not require the individual to carry the risk alone. The code supplies the reason to demand that, and it cannot deliver it. Portions of this answer are synthesised from the cited sources rather than quoted from any single one.

The first claim is compliance, and the unit does not dispute that a lawful notice can be built. The argument is that compliance and autonomy are different achievements. The paradox the team invokes is, on its best reading, a comparison between a stated valuation and an observed act collected by different instruments, which is why Kokolakis (2017) calls for evidence of actual behaviour rather than self-reported attitudes, and why Solove (2020) treats the paradox as largely a measurement artefact rather than a licence to assume that users do not care. The preference is unstable because perceived control over disclosure raises what people reveal, including information that identifies them (Brandimarte, Acquisti and Loewenstein 2012). The choice is unstable because the design decides the answer: in the field experiments of Utz and colleagues (2019), a binary choice drew more acceptance than a per-category mechanism, nudging toward a privacy-invasive default raised acceptance, and less than 0.1 per cent allowed cookies for all purposes when the default was privacy-protective. Two equal buttons record a click and cannot show that the flow was appropriate, which is the standard Nissenbaum (2004) sets, and a survey of smart home flows found advertising and indefinite storage least acceptable even where consent was given (Apthorpe and colleagues 2018). The repair on the unit's account is a privacy-protective default, purpose-limited flows, and a notice that explains purpose and duration rather than only the click. Portions of this answer are synthesised from the cited sources rather than quoted from any single one.

The team's first move is to treat principles as sufficient, and the unit's answer is that the regime is a list of named duties (UK Parliament 2018). A lawful basis must be identified before processing and documented (ICO 2024), and the notice it relies on answers only the transparency duty. The second move is to treat a notice as an explanation. Articles 13 to 15 require meaningful information about the logic involved, with the significance and the envisaged consequences of the processing (Article 29 Data Protection Working Party 2018), which is information about a system; a person refused by the tool wants to know why this decision happened to them, and that is the question the literature fights over, with Wachter, Mittelstadt and Floridi (2017) denying that the Regulation confers a right to it and Selbst and Powles (2017) reading Articles 13 to 15 as conferring it. Third, the review button is only a safeguard if it is real. The 2025 Act defines a decision as based solely on automated processing when there is no meaningful human involvement, and requires safeguards that give the person information, allow representations, human intervention and contest (UK Parliament 2025). A button whose outcome is unchanged by the person pressing it is not meaningful human involvement. Fourth, the impact assessment is where the team should record what it cannot say in a notice: the error rate, who carries it, and whether the score changes the order in which claims are read (ICO 2024). Finally, Edwards and Veale (2017) warn that a right to explanation is unlikely to remedy the harm that drives the demand for it, so the repair belongs in the design rather than in the wording. Portions of this answer are synthesised from the cited sources rather than quoted from any single one.

The note's first claim, that this is standard practice, is true and is not an answer: Mathur and colleagues (2019) found 1,818 instances across about 11,000 shopping sites, and Di Geronimo and colleagues (2020) found at least one dark pattern in 95 per cent of the mobile apps they analysed, so prevalence is the problem rather than a defence. The second claim, that disclosure repairs the design, is defeated by Bongard-Blanchy and colleagues (2021), who found that awareness of a manipulative design does not equip a person to resist it, and by Nouwens and colleagues (2020), whose field experiment showed that removing the opt-out button from the first page raised consent by 22 to 23 percentage points. Whether the design is unlawful depends on the contested question. The FTC (2022) would ask whether the countdown timer and the pre-selected plan deceive or unfairly harm a consumer, working from existing authority rather than a list. The Digital Services Act (European Parliament and Council 2022) would ask whether the interface deceives or manipulates recipients or impairs free and informed decisions, with Article 25(2) handing practices already covered by consumer or data protection law back to those instruments. Gray and colleagues (2021) warn that the design reading and the legal test are different kinds of thing, and Mathur, Kshirsagar and Mayer (2021) show that the term carries no single normative criterion, so appearing on a list proves nothing. What would settle it is evidence: the click-through and cancellation data the team already holds, and an enforcement decision or a Commission guideline under Article 25(3). Kollnig and colleagues (2022) show what a design requirement can change, and that it may relocate a practice rather than end it. Portions of this answer are synthesised from the cited sources rather than quoted from any single one.

The reviewer's two grounds are the outcome, that behaviour changed, and a lack of transparency, and the unit argues that neither is the criterion on its own. Susser, Roessler and Nissenbaum (2019) define online manipulation as covert influence that targets and exploits decision-making vulnerabilities, and Klenk (2020) defines it as influence whose means are not intended to reveal the user's reasons; Sunstein (2015) adds that a manipulative act is one that does not sufficiently engage reflective choice. Applied to the newsfeed, the ranking is tuned to capture attention, which Lorenz-Spreen and colleagues (2020) describe as the dominant design of the current ecosystem, and Yeung (2016) shows that such hypernudges are networked and continuously updated and that notice and consent cannot legitimise them; on the mechanism criterion the newsfeed is a candidate for manipulation. Applied to the fitness app, the reminders serve an end the person set, and the mechanism offers a reason to act rather than working around one, so it looks like persuasion, though Thaler and Sunstein (2008) would test it by whether the option to opt out is preserved. The contested positions separate here. The mechanism position would call the newsfeed manipulative even if nobody objected, and Wilkinson (2012) would ask whether the team intended to manipulate, adding that consent could make even a manipulation consistent with autonomy. Reported feeling is a weak test because a person can be influenced without objection and can resent an influence they can accept; Sunstein's criterion is whether reflective choice was engaged, not whether it was resented. What would settle it is evidence that the mechanism operates around the person's reasons: for the newsfeed, whether ranking choices are disclosed and whether the person can act on the disclosure; for the app, whether the reminders respect the user's stated goal and an easy opt-out. Calvo and colleagues (2020) point to the constructive test, which is whether the design supports human autonomy. Portions of this answer are synthesised from the cited sources rather than quoted from any single one.

The unit's thesis is that the ethical problem is the unequal relation between the observing party and the observed rather than the sensor, so the failures that matter are an unnamed lawful basis, a default retention period and an undisclosed reuse. The manager's first two grounds, consent and improved safety, do not meet the standard. The Information Commissioner's Office (2022) requires a lawful basis for monitoring and a documented retention schedule, so the absence of an explanation, the missing retention period and the sale to an insurance partner are three failures at once, and the guidance states that a customer making monitoring a condition of business does not justify it. Applying the contested positions separates the assessments. On the collection position, Moore (2000) would say the drivers' knowledge of the kinds of surveillance used should have been made explicit and that this evaluative monitoring violates their privacy rights; Bhave, Teo and Dalal (2019) would record the clash between the drivers' expectation of control and the firm's expectation of extensive access. On the use position, Ravid and colleagues (2019) and Kellogg, Valentine and Christin (2020) would ask what the scores do to the employment relation, and the undisclosed appraisal use is exactly the algorithmic control they describe. On the relation position, which is the unit's, Zuboff (2015, 2019) would read the sale of the scores as the clearest sign that the data is the product, and Marx (2002) would note that the scheme surveys a whole category of person rather than suspects. The manager's claim that compliance settles the ethics fails because the outcome evidence reports harm without a performance gain; Ravid, White and colleagues (2023) find raised stress that does not depend on the characteristics of the scheme, and Alkhatib and Bernstein (2019) show that the shift manager who acts on the score absorbs accountability the system produced. An instrument that would settle which position is right would vary disclosure, retention and reuse independently and measure what happens to the drivers under each, holding the rest of the scheme constant.

The unit's thesis is that fairness has no single definition, so choosing a metric is choosing which error will fall on whom; an audit can show that performance differs across groups but not that a system is fair, and the auditor's access is the binding limit. The vendor's parity claim is therefore a value choice presented as a verdict. Kleinberg, Mullainathan and Raghavan (2017) prove that three fairness conditions cannot all hold at once except in highly constrained cases, and Chouldechova (2017) shows the criteria conflict when prevalence differs across groups, so a model that satisfies demographic parity may simultaneously fail error rate balance; the flagging of disabled students at a higher rate is exactly the kind of group performance gap that parity does not rule out. Friedler, Scheidegger and Venkatasubramanian (2021) supply the reason, that the families of definition encode different value systems, so parity answers a question about representation while leaving open who bears the false positives and who bears the false negatives. A false positive here means a student wrongly flagged, with the stigma and intervention that follows, so the choice of metric decides who absorbs that cost. On access, Sandvig and colleagues (2014) show the code audit needs consent that is rarely granted, and the university withholds the data, so an outside audit would rely on output-based designs that support weaker causal inference; Metaxa and colleagues (2021) add that an audit measures a version of a system that changes silently. What would settle the question is a prospective study that holds the flagging task fixed, applies each family of definitions in turn, and measures both outcomes for disabled and other subgroups and the cost in overall accuracy, which is the trial Corbett-Davies and colleagues (2018) call for and the field has not run. Wang, Ramaswamy and Russakovsky (2022) add that the audit should report the intersection of attributes rather than one axis, since the disabled subgroup may itself be heterogeneous, and Pagano and colleagues (2023) note that single-metric, binary comparisons are the norm and that different metrics do not give uniform results.

The unit's thesis is that the Act attaches its obligations to the risk tier rather than to the principle, so the contested work moves into the definition of the system, and the accountability proposals that survive the evidence place a function inside the organisation rather than producing another document. The manager's first claim fails on the structure of the instrument. Regulation (EU) 2024/1689 prohibits a closed list of practices and attaches the heaviest duties, risk management, data governance, technical documentation, logging, human oversight and standards for accuracy and robustness, to systems it classifies as high risk, and credit scoring is named among the high-risk domains, so the question is which tier this system occupies rather than whether the team is in the UK. That is the definitional contest: Ebers and colleagues (2021) raise the scope of application and the high-risk provisions as points carrying problems, and Novelli, Casolari, Rotolo and colleagues (2023) show that the categories rest on broad fields of application, so the risk magnitude for a tool used both for lending and for retention offers may be estimated wrongly. Veale and Zuiderveen Borgesius (2021) add that the tier scheme is not self-executing, since some provisions are unclear and others may be ineffective. The market-access point does not help the manager; it is the reason the Act reaches the team at all, as Smuha (2021) frames in the competition between regulatory models and Bradford (2023) locates in the European rights-driven model, and where the Act arrives only through market access the voluntary NIST framework of Tabassi (2023) is the instrument that actually governs the team's risk process. The second claim fails because a data protection impact assessment is a document, and accountability is not a document. Novelli, Taddeo and Floridi (2023) define accountability as answerability with three conditions, authority recognition, interrogation and limitation of power, and two of those are institutional: a forum with standing to ask and a body able to act on the answer. Veale, Van Kleek and Binns (2018), interviewing twenty-seven public sector practitioners across five OECD countries, found a disconnect between practitioners' organisational and institutional realities and what research offered, which is the gap a document does not close. Against this, Selbst (2021) argues that documentation can still work if it is legible to the firm and feeds a cultural shift, so the assessment is not worthless, but even he concedes that institutional logics such as liability avoidance and the profit motive let firms undermine the mitigation goal. What would settle the disagreement is a prospective comparison of two lenders performing the same task, one with a standing accountability function holding authority, a forum and a limit on power, the other with documentation alone, measured on the harms the assessment is meant to prevent; the field has not run it, so the defensible position is that the document can support reporting but cannot supply the forum or the power.

The reply treats a passed audit as a discharge of the obligation, and this unit's sources show why that does not follow. The statutory duty is in section 20 of the Equality Act 2010, which requires a provider to take such steps as it is reasonable to have to take to avoid a substantial disadvantage caused by a provision, criterion or practice or by a physical feature, and to provide an auxiliary aid where one would remove the disadvantage (UK Parliament, 2010). The duty attaches to the council's practice and asks for steps in advance, so the arrival of one telephone call from one applicant is evidence about the practice rather than the trigger for the duty. Conformance is a different question. Power, Freire, Petrie and Swallow evaluated 16 websites with 32 blind users, recorded 1,383 instances of user problems, and found that only 50.4 per cent of them were covered by WCAG 2.0 success criteria and that in 16.7 per cent of websites the recommended technique had been applied while the problem remained (Power et al., 2012). WCAG 2.2 itself states that it will not address every user need and that criteria for cognitive, language and learning disabilities raised significant challenges of testability (W3C, 2023). The address list is thus neither a surprise nor a counsel of perfection: it is the ordinary residue that a standard leaves. Where the fix belongs is settled by the design literature. The social model locates the barrier in the arrangement rather than in the person, so the obligation attaches to the lookup rather than to the applicant's screen reader (Oliver, 2013), and the route from that premise to a specification is ability-based design, which asks what the user can do and adapts the system to it rather than to a diagnosis (Wobbrock et al., 2011). Shinohara and Wobbrock show what an audit cannot see, namely the social cost of needing an adaptation and the workarounds disabled people build for themselves (Shinohara and Wobbrock, 2011); the telephone call is that workaround, and treating it as an external exception is how the cost is moved from the design to the user. The evidence of success would not be a fresh audit pass but a measured change: the same task, attempted by disabled participants before and after the fix, with disabled people deciding what counts as success, which is the design Mack and colleagues identify as scarce in a field with a median participant group of 13 (Mack et al., 2021; Power et al., 2012). The council should also note that enforcement of this kind is moving towards a product regime with a conformity route and an exemption to argue, as the European Accessibility Act shows, so a claim that the matter can wait for a future release is now a claim against a live obligation (European Parliament and Council, 2019).

A strong answer starts by refusing to accept the method the client has named, and instead reconstructs the problem. The questions to put back are those that decide feasibility: who uses the app and how they will be reached; what decision the work must change; what existing evidence, analytics or support data could answer part of the question already; what is fixed and what is negotiable; and which date the answer is really needed by. With no build, a usability test is impossible, so the answer should name the decision the work informs. If the client is choosing between two directions, the appropriate method is formative, such as concept testing or task-based interviews on a prototype, not a summative test of a finished design. The deliverables should then be stated concretely: a research plan, a set of sessions to be run, and a findings report with a recommendation. The assumptions should be listed, including that the client supplies a clickable prototype by an agreed date and recruits from an accessible panel. The estimate should be built from comparable finished projects rather than from an imagined plan, because the research on the planning fallacy shows that plan-based estimates are systematically optimistic. The answer should refuse to promise a validated outcome measure or a predicted lift, because the method proposed cannot support that claim. Naming what will not be promised is a mark of scoping skill, not of weakness.

The first move is to gather evidence rather than assert an opinion, because the strongest critiques rest on something both sides can check. That means a keyboard or screen reader pass over the flow, notes on the specific steps where the control cannot be identified without colour, and a note of the accessibility requirement the design engages, such as the public sector duty to be perceivable, operable, understandable and robust. The conversation should separate the work from the person. A useful frame is to state the goal the flow was made for, ask what the colleague intended, and then present the passing and failing steps as observations rather than as a verdict on their ability. That approach follows the BCS Code of Conduct, which asks members to respect and value alternative viewpoints and to seek, accept and offer honest criticism of work: honest about the defect, respectful to the colleague. If the manager overrules the concern and instructs the team to ship, the professional question changes from craft to duty. The code places obligations on the individual, including a duty to consider the public interest and not to misrepresent or withhold information about the performance of a product. A strong answer records the concern, avoids signing off a claim that the flow is accessible, and escalates through a route the code supports, such as the individual's professional body or the organisation's own risk process. Accepting the instruction quietly is the one response the code does not easily permit.

The team lead is running the usability engineering conception the unit attributes to Nielsen (1993), where usability work is planned as lifecycle stages with measurable criteria, and the fixed early interface specification is the human-factors arrangement Bannon (1995) describes, in which ergonomics arrives near the end as a set of specifications. The product manager is operating inside the design lineage, where the argument is about what should be built: Fallman (2003) distinguishes research-oriented design from design-oriented research, and Dorst (2011) locates the designer's core practice in framing rather than in verifying a fixed specification. The graduate is proposing something whose status differs between those positions, so the first professional act is to read the setting rather than to repeat the proposal. Carroll (1997) explains why the graduate need not treat the lead as an opponent, since the field integrated scientific concerns with the engineering goal of usability, and Cooper (1999) describes the drift into building mechanics that the goal-directed design response was written against. To decide whether this is Harrison, Tatar and Sengers's (2007) paradigmatic clash or ordinary methodological breadth, the test the unit asks for is a comparison of consequences: would the same evidence be accepted under measurable utility, controlled contrastive demonstration and situated interpretation, and would two teams on the same problem diverge predictably in what they measure and build? Rogers (2012) suggests the paradigms may be borrowed theories put to different purposes, which is why the graduate should name the commitment in the room, agree in advance what result would change the decision, and argue inside the standard the team already accepts.

In the agency the graduate is working where the commission, the deadline and the audience come from the client rather than the discipline. The unit's hinge finding applies directly: Bak et al. (2008) found that resource demands and developer mindset, not ignorance of methods, were the main obstacles to usability work in practice, so a two-day evaluation answered in a week is a normal shape for this setting rather than a failure of rigour. The graduate's task includes educating the client, which is exactly what Buis et al. (2023) report practitioners doing with low-maturity clients across 20 case studies. On the short reporting cycle, Kujala et al. (2011) explain why the long-term instrument built for a product that lives for years is unavailable here, so the choice of method follows the setting rather than the graduate's preference. In the hospital trust the setting holds an explicit standard, so good work is work that can be shown to have followed the process, which is the position Earthy et al. (2001) describe for ISO 13407, now carried by ISO 9241-210:2019, and Gulliksen et al. (2003) describe in their twelve principles. The clinical safety process and the standards request are the setting's definition of evidence, and a designer who arrives with a research master's preference for exploratory depth must first negotiate how that group defines a good prototype, the tension Singh et al. (2025) document in cross-disciplinary work. To decide whether the graduate's difficulty is setting or skill, the unit asks for evidence the literature does not contain: a comparison of comparable projects in different settings, holding task and team roughly constant, measuring both the outcome and the practitioner's scope to act. The weaker check the graduate can run is whether the same person's measured impact changes when they move between the agency and the trust, which is the longitudinal test the unit names.

The client has supplied a proposed solution, a layout change, and treated it as the problem. The unit's first move is to separate the two. What is genuinely given is the situation: the helpline volume, the two-month window and the style guide are real constraints, and Schon (1983) does not license ignoring them; he describes framing a situation the client owns. What is questionable is the causal claim that the layout causes the calls. Lloyd and Scott (1994) found that the problem statement is a product of the design process rather than its input, so the honest brief-back treats 'the bill is badly laid out' as a hypothesis to test, not a finding. Dorst and Cross (2001) support the method: the problem and the solution co-evolve, so the project should develop rival framings and rival drafts together rather than settling the question first. A plausible rival frame is that customers do not understand a charge, not a layout: the same helpline volume could be driven by a confusing tariff or a missing explanation, which no layout change would fix. Restrepo and Christiaans (2004) explain why this frame should emerge from the work rather than be fixed up front, since designers' information needs follow the emerging structure of the problem. To test the frames I would combine two methods, because Goguen and Linde (1993) show each elicitation technique foregrounds different aspects of a setting: call-reason analysis of helpline records, and interviews with a small sample of customers reading their own bill. I would be explicit that Potts (1995) warns against importing a method built for a different kind of client. On measurement, the single helpline measure is defensible but insufficient, because a fall in calls could mean better understanding or simple abandonment. I would ask the client to accept a second measure tied to the frame, such as whether customers can correctly identify and explain a disputed charge after reading the bill, and I would record the framing decision so the choice is inspectable. If the client rejects the reframing, Paton and Dorst (2011) describe reframing as a situated negotiation, so the correct move is not to proceed silently with my own frame nor to abandon the question, but to record the client's frame as the governing one, state what evidence would change it, and let the call-reason analysis decide. Buxton (2007) gives the professional reason to have that argument in month one: getting the right design must precede getting the design right.

The first thing to say back to the account lead is that the plan is not evidence. Buehler, Griffin and Ross (1994) found that people underestimate their own completion times but not other people's, that predictions are built from imagined future scenarios rather than from comparable past jobs, and that the past is discounted as unrepresentative; in Study 4 the optimism disappeared only when participants were told to connect relevant past experience with the prediction. So the note should require the number to be built from the firm's own logged projects: hours actually spent on accessibility audits, remediation workshops and report writing, with the sample size stated. Where the firm's record is too thin, the class should be named and priced from published comparables, which is what Flyvbjerg (2006) sets out as reference class forecasting and what Flyvbjerg (2021) supports with base rates from 2,062 projects. The note should also record the range rather than a single date, because Little (2006) found estimation accuracy lognormally distributed with an uncertainty range that stayed nearly constant through a project, and because Jorgensen (2004) lists assessing the uncertainty of the estimate among the twelve best-practice guidelines. On the client conversation, the distinction is McConnell's (2006): the estimate is a prediction, the target is the client's desirable outcome, and the commitment is a promise to deliver defined scope at a quality level by a date. The account lead should present the range as a forecast, ask the client to own the target, and make any commitment in writing against a stated scope. Finally, the firm should log estimated against actual effort on this programme, because Jorgensen and Sjoberg (2001) found that estimates shape the work: projects under cost pressure trimmed the work to fit an optimistic figure, early anchors moved later estimates even when declared irrelevant, and optimistic estimates produced less effort and more errors. What could still go wrong is that the class may not be comparable, which is the caveat Baerenbold (2023) and Cantarelli and colleagues (2026) both raise, so the class definition should be written down and reviewed after delivery.

The note should open by saying that the contract is not paperwork around the work; it is where the assumptions, the ownership and the rights of reuse are allocated, and the proposal is the first place they can be seen (Raiffa, 1982, on structure before tactics). The legal point comes next: rights arise under law and begin with the maker rather than the payer, so silence does not leave the design files unowned, it leaves them with the team unless the client's template says otherwise (Bently and Sherman, 2014; Bently, Sherman, Gangjee and Johnson, 2026). Because Menell (2019) shows the law being revised rapidly around new technology, the team cannot rely on a default that happens to suit it now. The allocation to ask for is specific: the client takes the rights it needs to operate the service, the team keeps the right to reuse the patterns and to show the work, and any derived-work or training rights are named rather than left implicit. That is a position, not a courtesy, and the negotiation literature says to prepare it as one. Fisher and Ury (1981) would have the note distinguish the client's position (it wants to own everything) from its interest (it wants no surprises and no dependence), and Neale and Bazerman (1992) warn that the team's own frame, set on closing the deal, will bias how it reads the terms. Galinsky and colleagues (2008) predict that modelling the client's interests in the note will uncover agreements that sympathy alone would miss, and Curhan, Elfenbein and Xu (2006) say the relationship that survives the negotiation matters more to the next engagement than the fee does. Finally, the team should log estimated against agreed scope and record which clause each concern went into, because Maister (1993) locates the firm's economics in pricing and the client relationship together. What could still go wrong is that the client refuses and the team signs anyway, which is why the note should name the two or three clauses it will not concede and say what happens then.

The 211 per cent figure is real but it is not a causal result. Design Management Institute and Rae (2015) report that a market-capitalisation-weighted portfolio of sixteen design-led companies delivered ten-year returns 2.11 times the S&P 500 over 2005 to 2015, but the portfolio is hypothetical and its companies were selected on the same design practices the index is then used to credit. McKinsey and Company (2018) states plainly that its own index shows a correlation it cannot turn into causation, and reports that 300 companies were tracked rather than manipulated; its more usable findings are descriptive, that over 40 per cent were not talking to end users during development and just over 50 per cent had no objective way to assess design output. The reason to expect association rather than cause is in the method, since selecting winners on design practice and then reading their growth builds the relationship into the sample, and in the satisfaction literature, where Anderson, Fornell and Lehmann (1994) find the quality-to-profit link but also that market share can lower satisfaction, Anderson and Mittal (2000) show the links are asymmetric and nonlinear, and Rust and Zahorik (1993) show returns depend on the firm's position. The defensible alternative is a local estimate: Karat (1990) compared the dollar value of reduced end-user task time with the cost of the usability work and reported a savings-to-cost ratio of about 2 to 1 for a small project and 100 to 1 for a large one, which is precisely why the client's own project size and records must supply the number; Karat (2005) gives the form, translating reductions in user time and task error into a decision for managers, and Rajanen (2007) warns that the models' assumptions are rarely tested, so our estimate should be presented as an illustration with its assumptions named. Practically, I would price the change using the client's support logs, abandonment rates and time on task, use the two earlier internal trials as the local comparison, and cite Mijac, Jadric and Cukusic (2024) to show that user experience and co-creation are now accepted success factors in a validated model. To earn a causal claim later, the client would run a pre-registered two-arm test holding strategy and market constant, or pool several such tests, which is the logic Keiningham, Cooil, Andreassen and Aksoy (2007) applied when they failed to replicate the Net Promoter claim with 21 firms and more than 15,500 interviews. I would refuse to state that the 211 per cent figure proves redesign causes growth, and I would keep the word correlation in any slide that mentions it. Kimbell (2012) is the reason for one further caution: if value is produced across the organisation rather than by a design team alone, even a positive local result should be attributed carefully.

I would begin by naming the structure rather than the personalities. Carlile (2004) gives me the vocabulary: the engineer and the product manager are not short of information, they are answerable for consequences inside their own function, which makes this a pragmatic boundary where translation is not enough and each party's knowledge has to be transformed. Carlile (2002) explains why the resistance is not obstinacy, since knowledge is localised, embedded and invested in a function. Rittel and Webber (1973) and Buchanan (1992) set the expectation I will not pretend to exceed: a design problem of this kind has no definitive formulation and no stopping rule, so my test cannot show that the change is correct, only that one framing costs more than another. What the test can carry is stated carefully, in the form the unit recommends: most participants failed the same step, which is a reason to change a decision rather than a proof that the earlier decision was false. Before the next test I would ask the team to write two criteria down, what result changes the decision and what result leaves it alone, which is Rousseau's (2006) research-practice gap answered in a small way and Briner, Denyer and Rousseau's (2009) practitioner-side evidence-based management. I would record the options considered and the criteria, in the spirit of MacLean and colleagues' (1991) design space analysis, and note that they present the payback of such a record as a goal rather than a result. I would also expect the record itself to be used unevenly, since Pfeffer and Sutton (2006) find that practices depend on tacit knowledge and resist adoption through precedent, and I would present the case so that a steering group sees the method and not only the presenter, which Petty and Cacioppo (1986) predict will matter. What I would not claim is that the evidence proves the change is right, that the engineer is being unreasonable, or that the criterion we agree on is the correct criterion rather than a workable one.

I would begin where the loss actually happened, which is governance and knowledge rather than methods. For governance I would build a consent template and a record of what each participant agreed to, in the spirit of Kaplan's (2020) focus areas of participants and governance and Towsey's (2024) insistence that ethics and privacy be more than a checklist. For knowledge I would rebuild the findings into a searchable repository against the FAIR principles, that data be findable, accessible, interoperable and reusable and designed that way rather than patched afterwards (Wilkinson, Dumontier and Aalbersberg, 2016). I would reconstruct the participant panel lawfully, because a panel that cannot be re-contacted is not an asset. The test I would apply is not whether a thing exists but whether a colleague who was not present could find and reuse it, which is how Towsey (2024) justifies a knowledge management strategy. That test also tells me what is overhead: a tool nobody is obliged to use, or a report nobody can search, fails it. On the benefit I would be careful. Ives and Olson (1984) found the benefits of involvement not convincingly demonstrated and He and King (2008) found them minimal to moderate and stronger for attitudes than productivity, while Bano and Zowghi (2015) found the relationship neither direct nor binary. So I would promise continuity, lawful reuse and findable findings, and I would not promise that more research raises productivity. Because Martinelli and colleagues (2024) show small teams push research out unless something holds it in place, I would invest in the analysis step, where Kujala (2008) locates the value, and I would brief participants on their role rather than assume it, which is Damodaran's (1996) point. I would keep a record of each study's participation, its stage and its outcomes, following the separation of participation from involvement that Barki and Hartwick (1989, 1994) established, so the next researcher inherits evidence rather than anecdotes. What I would not claim is that the repository will improve the product, or that it will be used merely because it exists.

Begin with the claim the team cannot make. Removing names does not make the dataset anonymous: Ohm (2010) reviewed the re-identification research that defeated nominally anonymous datasets, so we state a measured re-identification risk rather than asserting anonymity, and we test the risk rather than assuming it. Second, the setting decides what reuse is defensible. Nissenbaum (2004) holds that information flows are judged by the norms of the context in which the information was shared, so the partner's new purpose is a change of context that needs a judgement rather than an administrative handshake, and Solove (2006) reminds us that privacy names several harms, so the briefing should say which harm the reuse risks. Third, consent is weak evidence of permission. Luger, Moran and Rodden (2013) found the forms through which consent is obtained frequently unreadable, and Acquisti, Brandimarte and Loewenstein (2015) found privacy preferences uncertain, context-dependent and malleable, so we return to participants rather than relying on a signature given for a different purpose. Fourth, the law constrains what we may hand on, and Peloquin and colleagues (2020) describe the difficulties the General Data Protection Regulation creates for secondary use, so the lawful basis is checked before anything is promised. What we may keep is the analysis and the record of what participants agreed to; what we may hand on is a dataset whose risk we have measured and whose reuse participants have been asked about; what we may not claim is that the transcripts are anonymous, or that the original consent already covers the new purpose.

Begin by separating the artefact from the person, because the evidence says this is not a nicety. Kluger and DeNisi (1996) found that more than a third of feedback interventions reduced performance and that the harm grows as attention moves closer to the self, so in twenty minutes I would talk about the artefact and the method and not about the designer's ability. Hattie and Timperley (2007) give the structure: where the work is going, how it is going, and where to next, at the level of the task and the process. Concretely, I would name the three defects, show the evidence behind each, and ask the junior what they were trying to achieve before saying whether it worked, which is the distinction Dannels and Martin (2008) imply when they show the studio critique socialising students into academic rather than professional identities. Second, I would change how the work is reviewed, not just this piece of it. Rather than a single late review, I would put a short critique at the point where the rationale is formed, because Fleischmann (2025) found peer critique under 90 students exchanged tacit as well as explicit knowledge and gave participants confidence, and because a standard that only a lead can apply is a standard the team cannot share. Third, I would treat the accessibility omission as a duty of my role, not a lapse of the junior's, since Lindberg and colleagues (2024) found design leaders regard a lack of awareness of design's ethical dimensions as a main problem and build structures to address it, and Marathe and Piper (2025) show the accessibility gap living in infrastructure rather than in a code. Over the quarter I would make the review criterion explicit before the work starts, so the junior and I can both check the same standard, and I would treat the arrangement as a claim to be tested rather than a fix assumed to have worked.

The response should be short and non-defensive. Acknowledge the request, restate what was agreed, and state clearly what delivering the feature would displace, since a fixed team working to a fixed deadline can add work only by removing it. Offer the two real options: deliver the agreed product on the agreed date, with the new feature logged as a candidate for a follow-on; or exchange it for something of comparable size, with the client accepting the consequence. Give a date by which the choice must be made, because an open offer that never closes is its own risk. Record the exchange in the decision log: the request, the date, the options, the client's decision and the reason. The scope agreement should have made this exchange routine rather than awkward, because a document that lists exclusions and a change route has already told the client that the boundary exists. The checkpoints should have caught the need earlier. A working slice shown at the previous checkpoint invites the client to notice a missing feature while there is still time to trade for it. The failure this scenario describes is usually not the late request; it is a team that treated the first two checkpoints as progress reports rather than as opportunities for the client to change its mind.

Start from what the scheme is for. A shared mark with no individual component rewards the member who contributes least, because their effort is anonymous and their benefit is identical. The design response follows from that mechanism: make contribution identifiable and attach a consequence to it. Use three layers rather than one. First, artefacts that already exist if the team works properly: the repository history, the decision log, the risk register and the records of the checkpoints. Second, a short peer assessment at mid-point and at the end, asking each member to rate the others on the six facets Hoegl and Gemuenden describe, with communication, coordination and balance of member contributions among them, so the instrument measures teamwork rather than popularity. Third, an individual portfolio in which each member claims their contribution and points to the artefact that evidences it. The weight should sit mostly on the group product, with a smaller individual adjustment, because an aggressively individualised mark teaches students to compete rather than to collaborate. Collect the evidence continuously rather than reconstructing it at the end, and show the mid-point ratings back to the team so it can correct. Three things will go wrong. Commit counts are a proxy that rewards volume and punishes documentation and review. Peer ratings carry friendship bias and the risk of retaliation within a small cohort. And a scheme that is not shown to students in advance is experienced as a trap. Name those risks in the reflective account rather than pretending the measure is objective.

Start with the structure of the agreement. It records only deliverables, which the unit says is the part written first and used least. Pohl (1994) explains the consequence: a list of features is an agreement artefact being read as a specification, so both week six and week ten become disputes about the truth of a sentence that only recorded what two parties accepted, and neither can be settled by argument. A minimal repair names what the product will not do, states the assumptions and dependencies the team does not control, and sets a change route that prices a request, names who accepts it and says what is dropped in exchange. Second, the estimate. Jorgensen (2004) finds expert estimation the dominant strategy with no substantial evidence favouring estimation models, and Flyvbjerg (2006) finds cost and demand forecasts systematically inaccurate through optimism bias and strategic misrepresentation, so the team should treat its original hours as a judgement to justify and compare against what comparable projects took, not as a measurement. Third, the baseline. The literature does not settle the contest: Beck et al. (2001) and Larman and Basili (2003) support evolving scope from practitioner reports and project history, the baseline position rests on forecast accuracy, and Boehm (2002) and Boehm and Turner (2003) hold that each approach has a home ground. Nothing the unit cites compares the two regimes on the usability of what is delivered. Practically, the team should renegotiate at week ten with an explicit exclusion list, a recorded assumption about what the client meant by reminders, and a priced change for the week six request, and should expect the recurring requirements problems that Mendez Fernandez et al. (2016) report rather than read them as its own failure. Portions of this answer are synthesised from the cited sources rather than quoted from any single one.

Start with the form of involvement. The team has supplied continuous contact without a bounded instrument, which the unit treats as the weak version. Hartwick and Barki (1994) separate participation, the behaviour, from involvement, the attitude, and find overall responsibility to be participation's critical dimension; a client who comments on everything but owns no part of the requirements occupies neither position in a way that predicts use. Steen (2013) frames co-design as joint inquiry and imagination rather than requirement gathering, and this project produced neither a shared record of what the client understood nor a correction of the team's misunderstanding. Second, the evidence. The literature does not support the expectation that presence alone improves the artefact: Ives and Olson (1984) found the early research methodologically flawed and the benefits unconvincingly demonstrated, He and King (2008) found participation across 82 studies minimally to moderately beneficial and stronger on attitudes than on productivity, and Bano and Zowghi (2015) found the relationship inconsistently measured. Kujala et al. (2005) found early involvement related to better requirements quality, but in a small single-country sample, and what distinguished the projects that gained was users acting as the information source rather than attending. What does support involvement is one comparison close to a controlled design: Trischler et al. (2017) found codesign concepts scoring higher in user benefit and novelty but lower in feasibility, and only in cohesive teams, which is also the unit's warning about design by committee. Third, the checkpoint the project lacked. It needed a date, a working slice to demonstrate, the acceptance criteria exercised in front of the client, and a recorded response with a named owner, with the client's reaction priced through the change route rather than absorbed silently. Practically, the team should now hold a reconstruction meeting in which the cancellation rules are written down and owned by a named person on each side, record the outstanding objections, and agree two remaining checkpoints, one of them exercising the cancellation rules in front of the client. The expectation to set is the one the evidence supports: a client who understands the constraints and accepts the result, not the assurance that one more round of comments will improve the artefact. Yu (2025) would add that this is involvement rather than participatory design in the strict sense, since no political commitment to empowerment is in play. Portions of this answer are synthesised from the cited sources rather than quoted from any single one.

The team has inverted the sequence the unit describes. Approval gates the order: approval first, then recruitment, then consent, then collection. By taking notes and recordings before submitting an application, the team has gathered data it cannot use and cannot treat as a pilot, because participants were contacted and data collected without an approved protocol (British Psychological Society, 2021). The claim that approval was unnecessary because nothing sensitive was discussed is a judgement the team is not entitled to make in advance: the code places responsibility for an adequate protocol with the researcher, which makes the team's own assessment the thing to be reviewed rather than a substitute for review. Second, the application would have forced the design to be settled early, requiring the team to state the purpose, the participants, what they are told, what data is held and what could go wrong. Third, on what approval can and cannot do: the Belmont Report states three principles, respect for persons, beneficence and justice, and concedes that they cannot always be applied so as to resolve particular problems beyond dispute (National Commission for the Protection of Human Subjects of Biomedical and Behavioural Research, 1979); the Declaration of Helsinki requires independent committee approval before research begins (World Medical Association, 2013); the Menlo Report adapts the framework for computing and adds respect for law and public interest (Kenneally and Dittrich, 2014). The HCI critics add that approval cannot make a study harmless and that difficulties surface in the encounter: Munteanu et al. (2015) find formal review poorly matched to field research with vulnerable users, and Waycott et al. (2017) record challenges that arise during studies. Fourth, the team should stop using the collected data, delete it, submit an application describing what it intends, and if it still wants early feedback obtain approval and valid consent first, treating approval as the beginning of an ethics that runs to the end of the project rather than a hurdle to clear (Bruckman, 2014).

The plan did its job in week six and can be credited for it: it recorded what was excluded, so the charity's request is a change rather than a surprise, which is the part of the classical apparatus most worth keeping for a small team (Project Management Institute, 2021). The missing register is the larger omission. Boehm (1991) requires that a risk be identified, prioritised, owned, mitigated and monitored at each review; the departure of a member and a late scope request were both foreseeable, and a register would have forced the team to name a trigger and a fallback before the week they were needed. Kutsch and Hall (2005) warn that managers typically deny, avoid, ignore and delay risk rather than manage it, so the team's reliance on talk would be the norm and not a defence. The missing log matters because the decision was made and then lost. Kerth (2001) makes the retrospective a facilitated review whose purpose is to turn a project's history into a change, and Andriyani, Hoda and Amor (2017) show that teams may not reach all levels of reflection simply by holding such a meeting, so a decision that is not written down and tested cannot be relied on. The team should expect a fair hearing on the counter-argument: DeMarco and Lister (1987) argue that the major problems of the work are sociological rather than technological, Kalenda, Hyna and Rossi (2018) find culture and management support decisive rather than the scheme, and Dingsoyr and colleagues (2012) note that agile practices are claimed efficacious while empirical validation is lacking. The unit's answer is that the artefacts are cheap, they make decisions visible, and no study has shown a smaller set to do the same work. Concretely: add three live register entries with owners, triggers and dates; start a log entry for the week six decision, recording what was decided, on what information, and what was rejected; and put both on the agenda of the next weekly review.

The single mainline and the verifying build exist to find an integration error while it is still cheap to fix. Fowler (2024) defines the practice as each member merging into a shared codebase together with colleagues' changes at least daily, every integration verified by an automated build that includes tests, and the practices he lists include fixing a broken build immediately and keeping the build fast. The team's week-long broken build is the failure that rule is written against. Where the integration cost goes is documented: Gousios, Pinzger and van Deursen (2014) found that the pull-based model offers fast turnaround and a shorter time to incorporate a contribution, with mostly non-technical reasons for rejection, so branching does not remove the cost, it moves it into review and waiting. Beller, Gousios and Zaidman (2017) analysed 2,640,825 builds and found testing the single most important reason builds fail and that the build service is not an adequate substitute for local testing, which is why a team without tests gets no benefit from the build at all. The team's counter-argument has real support: Fowler (2024) himself names a team without a strong test suite and a project whose contributors are not full time as the conditions under which continuous integration fits poorly, and the unit adds that a part-time student team with an immature test suite resembles that open-source condition. But the remedy that follows is to build the missing conditions, not to keep branching. The team should commit to one mainline, write a small automated test suite so the build can fail meaningfully, fix a broken build the same day, and reserve branches for work that cannot be split.

The rule the unit defends is thin but absolute: no change reaches the mainline without a second reader, because the reading is cheap, it makes the change visible to someone who did not write it, and no study has shown a smaller commitment to do the same work. The decay is evidence for the rule rather than against it. Two bugs reached the client because no reader looked, which is the outcome McIntosh and colleagues (2014) associated with low review coverage: they estimated that low coverage produces components with up to two additional post-release defects, and Kemerer and Paulk (2009) found review rate a significant factor in defect removal effectiveness, with more careful reviews finding more defects. The member who approves in minutes is running the performance mode the unit names: where review is also where impressions form (Bosu et al., 2017), a fast approval can satisfy the rule while the reading stays shallow. The unreviewed module is a different failure, because the single reader is also the single expert, and Thongtanunam and colleagues (2016) showed that review attention is patterned rather than evenly spread, so the team must rotate the second reader instead of relying on the one member who understands the part. Dropping review would remove the only mechanism the evidence supports, and it would also remove the knowledge benefit the team is already underusing, since review spreads the files a developer knows about (Rigby and Bird, 2013) and carries knowledge transfer as well as defects (Bacchelli and Bird, 2013). The team should keep the thin rule and make it real: set a modest reading-time or coverage expectation rather than a heavy process, rotate reviewers so expertise spreads, and record what a review caught. It should also be candid that the sources are correlational and, in the richest case, single-company (Sadowski et al., 2018), so the rule is a defensible minimum rather than a proven cause of the client's two bugs.

The scheme is not fair, and the unit's argument explains why without appealing to sentiment. The dashboard counts proxies, and every proxy it uses prices the quiet work at zero: Maya's documentation, decision log and reviews leave few commits, which is exactly the work that keeps a project maintainable but that a commit count ignores. The evidence for treating contribution as part of quality is solid, since Hoegl and Gemuenden (2001) found balance of member contributions to be one facet of a six-facet model of teamwork quality that predicted performance across 145 German software teams, and Latane, Williams and Harkins (1979), extended by the 78-study meta-analysis of Karau and Williams (1993), show that effort does fall as groups grow and that identifiability is the lever. But identifiability is not the same as counting output. The instruments built to measure contribution rate behaviour rather than volume: Loughry, Ohland and Moore (2007) developed 87 behaviourally anchored items across five categories of team-member effectiveness, and Kaufman, Felder and Fuller (2000) show such ratings feeding a weighting factor, so the tool the module already resembles measures responsibility rather than commits. The dashboard's error is also one of resolution. Falchikov and Goldfinch (2000) found peer and teacher marks agree more closely for global judgements than for multi-dimensional ones, and a contribution share is multi-dimensional, while Topping (1998) found 18 of 31 reliability studies adequate but seven unacceptably low, so a single share is the thinnest case for the instrument. And raters cannot see what they rate: Freeman and Greenacre (2010) found that actively contributing students had difficulty distinguishing teammates who were socially loafing from those genuinely struggling, so Maya's low commit count is indistinguishable from slackness on the dashboard. What should change is the design, not Maya's share. The module should gather behaviourally anchored peer and self ratings alongside repository evidence, weight them at a team level rather than by lines changed, and give a documented route by which a member can contest the reading, because the unit's defensible aim is a fair process rather than a number that claims to be fair.

The single score supports less than it appears to. Its first weakness is the observer: Hoegl and Gemuenden (2001) found that the strength of the teamwork-quality association with performance varied with who rated the performance, and Lindsjorn and colleagues (2016) found a positive effect when team members and team leaders rated it but a negligible effect when product owners rated it, so a score drawn only from the team is one perspective, and the client's contrary view is evidence rather than noise. Its second weakness is the construct: psychological safety is regularly conflated with trust and cohesion (Edmondson and Lei, 2014), so a survey showing that a team feels it is working well cannot distinguish comfort from safety. The member who stopped speaking is the case that matters, because psychological safety is defined as a shared belief that the team is safe for interpersonal risk taking, and it predicted learning behaviour in the founding study (Edmondson, 1999). The module should report the direction and the levers rather than one number: it should date the coordination checkpoints at the boundaries between episodes of work (Marks et al., 2001), record how the team handled the week-five disagreement against the three resolution tendencies, focusing on content rather than delivery style, stating the reasons behind work assignments, and assigning by task expertise (Behfar et al., 2008), and treat the client's view as a second observer with a named perspective. What the literature cannot supply is a validated score for the module to report, since the fifty-year review of team research still lists better measurement among its open challenges (Salas et al., 2008).

The stranger test comes first, because it decides what the final week is for. The handover succeeds only if a competent maintainer can run and modify the product from the repository alone, and maintainability has a standard form, the ability of intended maintainers to modify the product with effectiveness and efficiency, decomposed into modularity, reusability, analysability, modifiability and testability (ISO, 2011). A repository a maintainer cannot analyse, modify or test has not been handed over, however good the closing presentation was, and Lehman (1980) shows why this matters: maintenance is a never-ending activity, so the product's real quality is spent after the team leaves. The evidence then bears on how to spend the week. Formal prose is what developers consult last and repair least, since Forward and Lethbridge (2002) found documentation maintenance rarely happened and professionals wanted tools extracting knowledge from code and tests, Lethbridge, Singer and Forward (2003) found engineers update documentation less than advisers assume, and Aghajani and colleagues (2020) found documentation valued in principle but inadequate, obsolete and ambiguous in practice. A 40-page document is therefore a poor bet, but abandoning the record is also wrong, because the design rationale is exactly what the code cannot recover (Parnas and Clements, 1986). The remaining week should buy a decision log of the choices a stranger cannot infer, tests that pin the behaviour down, and names and structure a maintainer can follow, with only the minimum prose that carries the decisions. The strongest objection is that the evidence only shows documentation is imperfect, not that it is dispensable: outdated documentation remains useful, and self-explaining code is not self-maintaining, because comments drift out of step with the code (Wen et al., 2019), so the disciplined answer might be targeted and maintained documentation rather than less of it. What would settle the disagreement is a controlled handover study comparing a prose specification, a self-explaining repository, and both, measured on the time to the first correct change after the team leaves; no such study exists, which is why the advice is bounded.

The first test is honesty about what the team did. Argyris and Schon (1974) distinguish espoused theory from theory-in-use, so an account that asserts learning without naming what the team actually did is espoused theory, and the decision log is exactly the record that can contradict it; Dewey (1933) requires each lesson to be held up to the grounds that support it rather than merely asserted. The second move is to write the account against that evidence rather than against the term's calendar, because Geirsdotter Baekkelund and colleagues (2025) found capstone teams defined the problem late and iterated mainly when mentors pushed them, which means an unaided narrative tends to rationalise the process it describes. The drafted conclusion should be replaced with rules: what the team will do differently about a change of scope, and what it will do when a disagreement about testing is unresolved, each specific enough to be shown wrong, which is what Kolb (1984) and Gibbs (1988) intend by turning an experience into a claim for the next attempt. The strongest objection is that the evidence behind reflective practice is largely theoretical and its measures inconsistent, so rewriting for depth may only produce a better performance of reflection (Mann et al., 2007). The team should answer that objection by making the account testable rather than more eloquent and by naming the gap: no study yet measures whether stating a rule changes later behaviour. What would settle the disagreement is a study that scores the rule an account states and then measures the writer's behaviour on a later task against it.

Begin by naming the three parts precisely. The claim is that the reminder improves adherence. The evidence is 24 patients observed for four weeks, with the effect size the authors report. The inference is the step from those 24 people over four weeks to the general claim that reminders improve adherence, and that step is wide. It crosses two gaps. The first is the sample: 24 participants cannot detect a small effect, and a large estimate from a small sample tends to exaggerate the true size. The second is the window: four weeks cannot show whether the effect survives the novelty of a new device. The two studies that would strengthen the work are a preregistered replication in a larger sample, which tests whether the effect is real, and a longer deployment, which tests whether the effect persists as the reminder becomes ordinary. A replication addresses the sampling gap and a deployment addresses the duration gap, and they answer different questions, so neither substitutes for the other. The result that would change your mind is a preregistered replication that finds no reliable difference, or one that finds an effect far smaller than the original. Either would suggest the first study overestimated the benefit, which is common in small samples. Note the boundary of your own reading: if the paper reported its analysis as exploratory, the problem is labelling rather than the finding.

Organise by argument, not by chronology. A weaker version lists studies of older adults and studies of generative interfaces one after another. A stronger version states the contested question first: whether generative output helps older adults complete a task, or whether its unpredictability raises the cost of error for users who are least willing to risk one. Then group the evidence by what it supports. One group reports benefits in controlled tasks. A second group reports usability and trust problems that appear in use. A third group turns out to be position papers and design guidance, such as the principles for generative applications, which propose a vocabulary but test no outcome. Naming that third group as guidance rather than evidence is the position your review takes, and it makes your own project's gap visible. For the search, write the question and protocol before searching. Record the databases, the search string and the date. Report the counts at each stage: records identified, records after duplicates removed, records screened on title and abstract, full texts assessed, and studies included. Under PRISMA 2020 you would also appraise the risk of bias in each included study and state how results were synthesised. The limitation to state plainly is the count itself. If three studies include older adults, the review cannot support a strong conclusion about them, and saying so is more honest than generalising from other populations.

The student asserts the first position in the contested section without arguing for it: Wobbrock and Kientz (2016) report that each type implies how it is judged, so declaring the type is an argument about the claim rather than a description of the method. The seven types are empirical, artefact, methodological, theoretical, benchmark or dataset, survey and opinion, and each faces a different examination: a finding on its importance and the soundness of the method, an artefact on what it makes possible, a method on utility, a theory on explanatory power, a dataset on the usefulness of its corpus, a survey on its organisation of what is known, and an opinion piece on the strength of its argument. Soundness answers only the first. If the student's study is empirical, the second review question is whether the finding matters, which the methods section cannot answer. The design literature attacks the claim from a different direction, since Fallman (2003), Zimmerman, Forlizzi and Evenson (2007) and Hook and Lowgren (2012) argue that making produces knowledge at an intermediate level that the empirical test misjudges; Gaver (2012) adds that research through design yields theories that are provisional, contingent and aspirational rather than tested and general. The CHI 2016 distribution shows that type carried information about acceptance, with empirical studies of people accepted at 26.3 percent and datasets at 16.7 percent, but Wobbrock and Kientz warn that activity is not value, and the student is treating a submission statistic as a judgement of worth. Sutcliffe (2000) and Gray, Stolterman and Siegel (2014) supply the missing test, which is whether another person can use the result. Evidence that would make the claim good is a report of what the contribution enabled: who reused it, for what, and what changed, together with a stated and defended account of the kind of claim the paper makes. Portions of this answer are synthesised from the cited sources rather than quoted from any single one.

The reviewer has diagnosed one cause of the claim-evidence gap and has mistaken it for the whole. Spin is a real and measured practice. Boutron and colleagues (2010) examined 616 published reports of randomised controlled trials, found 72 with statistically nonsignificant primary outcomes, and coded each for spin, which they define as reporting strategies that highlight a treatment as beneficial despite a nonsignificant difference for the primary outcome. Spin appeared in 13 titles (18.0 per cent), in the results section of 27 abstracts (37.5 per cent) and in the conclusions of 42 abstracts (58.3 per cent), so the reviewer is right that rewriting is sometimes the remedy. But two other mechanisms are outside the abstract. The first is design and analysis: Button and colleagues (2013) estimated the median power of neuroscience studies at between about 8 and about 31 per cent, which means a significant result from a small study is more likely to overstate the effect, and Ioannidis (2005) modelled the probability that a published claim is true as a function of power, bias and the ratio of true to false relationships tested, finding it can fall below one half. No careful sentence can make an overestimated effect honest. The second is misreading of the statistics that are reported: Hoekstra and colleagues (2014) found that 120 researchers and 442 students each endorsed more than three false statements about a confidence interval, and that experience with statistics did not help, so the gap is also produced when a correctly reported number is read as something it is not. The reviewer takes the position that the gap is made in the writing, the position Boutron's audit supports. The opposing position holds that the gap is in the statistics, and it is held by Greenland and colleagues (2016), who argue that correct interpretation taxes working scientists beyond what they can spare, and by Button and colleagues (2013) and Ioannidis (2005), who locate it in design and analysis. Wasserstein and Lazar (2016) and Nickerson (2000) sit between, holding that the tools can be used well when applied with judgement. Evidence that would settle the reviewer's claim is a corpus study that has not been run: have readers who did not write the papers re-derive each inference from the data alone, and code every departure of the stated claim for whether it follows from a reporting feature, such as spin, or a design feature, such as low power. If departures cluster in low-powered studies, the reviewer's rewrite is necessary but insufficient.

The editor has run together three claims the unit keeps apart. The unit holds that a venue is not a container for a finding but the community that decides what a finding is worth, a judgement made by named people under stated rules and moved by prestige, blinding and incentives (Liu et al., 2014; Tennant et al., 2017). The controlled prestige studies show that author standing reaches the verdict even when the work is identical. Okike and colleagues (2016) sent the same fabricated manuscript to 119 reviewers and found about 87 per cent recommended acceptance when prestigious names were visible against about 68 per cent when they were not, with no difference in errors detected, and Tomkins, Zhang and Heavlin (2017) found single-blind reviewers significantly more likely to recommend acceptance for famous authors and top institutions, with odds multipliers of 1.63, 1.58 and 2.10. Double-blind review reduces but does not remove that advantage. The two blinding trials measured something else: whether hiding identity improves the general quality of written reviews. van Rooyen and colleagues (1998) found quality scores of 2.82 against 2.96 and 2.87 against 2.90 with no significant difference in recommendations or time, and Godlee, Gale and Martyn (1998) found no significant difference in the weaknesses detected, with blinded reviewers if anything less likely to recommend rejection. A null result for quality is therefore not evidence that identity is irrelevant. The editor also confuses bias with reliability. Bornmann (2011) reports that inter-reviewer agreement is generally low, which is a reliability problem, while Lee and colleagues (2013) argue the bias evidence has methodological limits; low agreement does not show the disagreement is unbiased. What would settle the matter is a trial in which real submissions are randomly assigned to blinded and unblinded review and compared for papers whose quality is estimated independently, so bias and noise can be separated.

The colleague has run together three claims the unit keeps apart. First, a single study records one observation under one set of conditions, and the step from that observation to a general claim about touchscreen users is exactly what a replication tests; a finding confirmed once is a hypothesis with a result attached (Nosek and Errington, 2020). The audit of HCI is stark: Hornbaek and colleagues (2014) examined 891 papers and found that 3 per cent attempted replication, and those replications rarely compared effect sizes, so an 18-student study cannot settle an effect. Second, the neighbouring audits show that effects usually shrink when re-estimated and that a substantial minority fail: the Open Science Collaboration (2015) found that 97 per cent of originals were significant against 36 per cent of replications with effects halved, and Camerer and colleagues (2018) found 13 of 21, or 62 per cent, replicating with effect sizes about half the original. Third, a same-direction result with a different measure is a conceptual replication, which Nosek and Errington (2020) say is informative only when the outcome is diagnostic evidence about a stated claim; error rate and time are not interchangeable, so the second laboratory has added weak evidence rather than confirmed the first. The strongest counter-argument is that HCI's design claims are about what is possible and so do not need general laws (Camerer et al., 2018), but the unit replies that this changes the scope of the claim rather than defeating it. What would settle it is a pre-registered multi-site protocol that compares effect sizes across samples rather than recording a confirm or reject decision (Klein et al., 2018).

The colleague has taken three degrees of freedom without recording them. Wicherts and colleagues (2016) catalogue 34 decisions open to a researcher and argue that opportunistically using them inflates false positives and effect sizes; dropping two measures after seeing that nothing was significant, and then re-running the analysis, is exactly the selection of the measure and the subset the checklist is meant to fix in advance. Reporting the surviving measure as though it had been the hypothesis is HARKing, the practice Kerr (1998) named, and it matters because the reader can no longer tell which decisions were planned and which were chosen once the pattern was visible. The choices are not unusual: John, Loewenstein and Prelec (2012) found self-admission rates of 63.4 per cent for failing to report all dependent measures and 55.9 per cent for deciding whether to collect more data after checking significance, and Fanelli (2009) placed these lesser practices on a continuum with the 1.97 per cent who admitted fabricating or falsifying data. The aggregate evidence supports the colleague's sceptic: Head and colleagues (2015) found an excess of p values just below 0.05 across the literature, and Franco, Malhotra and Simonovits (2014) found that only 10 of 48 null results were published against 56 of 91 strongly significant results, so null results are the ones that vanish. Head and colleagues also limit the thesis, concluding that p-hacking's effect seems weak relative to the real effect sizes measured. Replacing the significance test would not help, because Bishop (2019) shows the four sources of irreproducibility each need their own remedy, and a new statistic does not stop the researcher choosing the measure after the data arrive. What would settle it is reporting the whole set of defensible analyses, as Dragicevic and colleagues' (2019) multiverse reports invite, so a reader can see whether the conclusion depends on the two measures that were kept.

A defensible answer separates what the remedies can do from what they cannot. Preregistration fixes a plan before the data are seen, so that planned prediction can be told apart from postdiction (Nosek et al., 2018), and the registered report goes further by having a journal accept the method before collection, so publication no longer depends on the result (Chambers, 2013). The evidence that they work is real. Kidwell and colleagues (2016) found that reported open data in Psychological Science rose from under 3 per cent to 23 per cent, and to 39 per cent in early 2015, after badges were offered, with no change in the comparison journals, and Chambers and Tzavella (2022) report early evidence that registered reports work as intended. The evidence that they work only partly is also real. Claesen and colleagues (2021) found that most of the 27 badge-holding studies departed from their plans and that nine disclosed none of the departures, and Hardwicke and colleagues (2018) found that although a mandatory open data policy raised availability statements from 25 to 78 per cent, 64 of 1,324 re-checked values could not be reproduced. The critic's argument is Szollosi and colleagues' (2020): the diagnosticity of a test depends on how well the statistical model maps onto the theory, and registration does not improve that mapping. The cost falls on early-career researchers, who report missing incentives, cultural resistance and intellectual-property concern (Chakravorti et al., 2025) against the flexibility and time the practices demand (Allen and Mehler, 2019). The honest position is to preregister the confirmation and to label the exploration as exploration. What would settle the dispute is evidence on adherence and on whether preregistered work reaches more defensible conclusions.

A defensible answer starts with what separates a method from a reading list. A systematic review fixes its question, databases, search strings, inclusion criteria and appraisal in advance and reports the counts at each stage, so Moher and colleagues (2009) introduced the four-phase flow diagram from records identified through screening and eligibility to the studies included, and Page and colleagues (2021a) publish PRISMA 2020 with a twenty seven item checklist and revised flow diagrams. The question then chooses the type: a scoping review suits identifying gaps, mapping a literature, clarifying concepts or examining research conduct, while a systematic review suits a narrow question evidence can settle, and the type must be fixed before the search is designed because it sets the retrieval requirements (Munn et al., 2018; Sutton et al., 2019). Scoping reviews have their own standard, PRISMA-ScR, with twenty essential reporting items and two optional ones (Tricco et al., 2018), so the group's forty-paper table may be legitimate as a map, provided it is labelled and reported as one. Pooling is a design decision: Higgins and Thompson (2002) define I-squared as the proportion of total variation in study estimates due to heterogeneity rather than chance and argue it should be reported in preference to a test for heterogeneity, Pigott and Polanin (2020) require explicit effect-size coding, attention to dependence among effects from one study and reproducible reporting, and Gurevitch and colleagues (2018) record that pooling studies that measure different constructs is where synthesis draws its criticism. Hertzum's (2026) meta-analysis of 105 studies found the System Usability Scale strongly correlated with self-reported workload but only weakly related to performance, which is exactly the kind of limit that pooling establishes and a reading list cannot. The contested question is whether the genre as practised is trustworthy: Ioannidis (2016) reports 266,782 items tagged as systematic reviews and 58,611 as meta-analyses between 1986 and 2015, annual growth of 2,728 and 2,635 per cent between 1991 and 2014 against 153 per cent for all indexed items, and concludes that possibly the large majority are unnecessary, misleading or conflicted, while Page and colleagues, Tricco and colleagues, Munn and colleagues and Pigott and Polanin treat the failures as failures of conduct against a specification that exists. What would settle it is a prospective audit of registered protocols, a count of duplicating reviews, and evidence that a synthesis changed a decision. The recommendation follows: the group should either commit to the narrower question with a protocol fixed before the search, or keep the forty papers and publish them honestly as a scoping review with a stated question and reported counts. Combining the breadth of a reading list with the label of a systematic review is the one option the literature does not license.

A defensible answer starts with the genre as a commitment. Grant and Booth (2009) analysed fourteen review types and found that few possess a prescribed and explicit methodology and that many fall short of being mutually exclusive, so calling a paper integrative promises a method the group has not yet specified. The fifty-paper summary fails on two counts at once. On Whetten's (1989) two questions, what is new and so what, a collection of summaries is new in no way and changes no practice, and Whetten states plainly that relationships, not lists, are the domain of theory. Sutton and Staw (1995) list references, data, variables, diagrams and hypotheses as the things offered in place of theory, and theory is the explanation of why a relation holds, so the group must supply the explanation for when gaze contingency helps. The second proposal therefore has the right shape, but the genre should match the question. Snyder (2019) argues that the research question and the purpose choose among the systematic, semi-systematic and integrative approaches, and an integrative review exists so that a new framework can emerge from work that has been conceptualised differently across traditions; her warning is that reviews labelled integrative are frequently summaries and not truly integrative. If the group's question is when contingency helps, an integrative or semi-systematic review is defensible and a narrow systematic review probably is not, because the studies differ in task, display and measure. Building the position has a procedure: Wolfswinkel and colleagues (2013) adapt grounded theory to the review as a five-stage process that runs from the protocol through coding to a conceptual model, and it produces theory from a corpus rather than a summary of it. The idea should be typed in Gregor's (2006) terms, since her five theory types, for analysing, explaining, predicting, explaining and predicting, and design and action, are judged differently, and Shepherd and Suddaby (2017) give the tools for building it. A reader judges the result with Boote and Beile's (2005) twelve-item rubric in five categories, coverage, synthesis, methodology, significance and rhetoric; their audit of dissertations found mean quality 2.09 on a three-point scale and low average scores on the synthesis criteria, so the group should expect the organising idea, not the search, to be the hard part. Where the literature disagrees is whether a standard can supply that idea, and the evidence that would settle it is a reliability study of whether reviewers agree on whether a review gained a new perspective, and a follow-up study of whether later work cites a review's framework as a claim to test. My recommendation is to fix the question first, label the genre honestly, and state the organising idea as a claim the corpus could disconfirm.

The team has confused a synthesis with a test. Weisz and colleagues (2024) derived six principles from a literature review, practitioner feedback and validation against existing applications, and used them in two applications; that is evidence about a principle's usefulness as guidance, not evidence of an outcome for a user. Amershi and colleagues (2019) are equally candid, validating eighteen guidelines with 49 practitioners against 20 products without a comparison application scored on an outcome. So the team's claim, that the assistant will help, is a hypothesis with no outcome attached, and the field's most cited design documents carry the same gap. Two measured results show why an untested design can hurt. Bansal and colleagues (2021) found across three datasets that explanations raised the chance a person accepted the recommendation whether or not it was correct, so a feature assumed to help changed trust without improving accuracy. Bucinca and colleagues (2021), with 199 participants, found that cognitive forcing functions significantly reduced overreliance but that the designs which reduced it most drew the least favourable subjective ratings, a trade-off on an axis the team is not measuring. Dell'Acqua and colleagues (2023), in a preregistered field experiment with 758 workers, found gains inside the model's capabilities and 19 percentage points fewer correct answers on a task outside them. The team should therefore name an outcome for a stated user, compare a group using the assistant with a group doing the same writing task without it, and fix the outcome, the sample and the analysis before the data arrive. The strongest objection is the one the unit answers: because Bansal and colleagues and Bucinca and colleagues each found one measure improving while another worsened, no single comparison settles a principle, so the team could argue that a principle is the better unit. The reply is that the principle must name the outcome it claims to improve and the cost it accepts; one that names neither cannot be contradicted and so cannot be learned from. Tabone and de Winter (2023) make the closing point, arguing that a new tool's validity must be established task by task before it is used. Lee and colleagues (2025) show the same limit in a self-reported, cross-sectional study that can establish an association but not a causal direction.

The announcement bundles three things, and the audit begins by separating them. The measured part is the pilot result from 40 households during peak hours, and a pilot of that size measures the households it contains rather than the population, a limit the unit's field-trial audit makes explicit: Brown, Reeves and Sherwood (2011) document demand characteristics, the interdependence of how a trial is run and what it produces, and dependence of a result on a subset of participants, all of which a 40-household deployment must address before its result can carry the claim. The survey measures stated concern, not behaviour, and the survey is not evidence about the automatic system at all. The projected part is the claim that automation will make sustainable behaviour effortless, which is a projection in the sense the unit defines: it names a future the evidence does not yet reach. The three audits then apply. On material cost, Freitag and colleagues (2021) put ICT's share of global greenhouse gas emissions at 1.8 to 2.8 per cent, rising to 2.1 to 3.9 per cent once supply-chain truncation is corrected, so any automatic system that adds networked devices adds to a measured cost that the announcement does not mention. On distortion, the pilot must bound the effects Brown and colleagues (2011) name rather than treating the field setting as self-validating. On inheritance, the claim that automation makes sustainable behaviour effortless repeats a lineage the field has audited before: Blevis (2007) framed design as choosing among future ways of being with a material rubric of use, reuse and disposal, and Brynjarsdottir and colleagues (2012) argued that persuasion narrows sustainability to individual optimisation and places technology as an objective arbiter over complex issues. An effort-free automatic system is that narrowing in a new form. The contested question is whether this critique is itself evidence. The measurement camp would say the audit earns its standing only where it attaches a number, and Freitag and colleagues supply one; the interpretive camp, following Dourish and Mainwaring (2012), would say the audit's work is to expose an inheritance the announcement did not inspect, and Nicenboim and Oogjes (2025) show that such a critique can become method rather than opinion. The strongest objection to my audit is that demanding a number may repeat the totalising move it criticises, since Dourish and Mainwaring link the drive to unite scattered elements under one scheme to a colonial inheritance. The answer is that a declared scope is the opposite of a totalising metric: Freitag and colleagues state which pathways they added, so the figure can be contradicted, and an audit that names what it excludes stays refutable. My advice is therefore to keep the pilot, drop the survey from the announcement, state the material cost the system adds, bound the field distortions, and describe the effortless-behaviour claim as a projection to be tested rather than a finding.

A strong answer makes each design choice explicit and defends it against a named alternative. For sampling, maximum variation across the two settings answers a comparative question, while a homogeneous sample inside one service answers a depth question about that service. If the aim is to find what holds across both, maximum variation is defensible. If the aim is to understand one service well enough to improve it, maximum variation is the wrong choice, and a criterion or typical case sample serves the question better. For recruitment, snowball sampling through clinic staff finds people who know the service, but it also finds people connected to those staff, which narrows the range of experience. Name that cost and say how you will counter it. For stopping, saturation is a claim about your own data rather than a number borrowed from a review. The range of 9 to 17 interviews reported by Hennink and Kaiser held particularly for homogeneous populations with narrow objectives, and this study is neither. State the point at which new data stopped adding codes or meanings, and describe the features of your sample that make the decision uncertain. For reporting, give the number, the criterion you applied, and the disconfirming cases you kept. A study that reports only the cases that fit its emerging pattern is weaker than one that names the case that does not.

The design choice has to follow the question rather than fashion. If the aim is to explain a measured effect, explanatory sequential is the fit: run the quantitative comparison of appointment attendance first, then interview a purposive sample of people whose behaviour diverged from the average. If the aim is to build an instrument for a population whose concerns are unknown, exploratory sequential is the fit, with interviews first and a questionnaire built from what they produce. If the aim is to see whether a measured change and a reported change agree, convergent is the fit, with the strands running in parallel. The integration point has to be named before collection. For explanatory sequential it is the sampling of interview participants from the quantitative results. The joint display puts a theme beside the figure it explains. One row might set a ten per cent rise in attendance beside the theme that reminders removed the need to remember, with a third column holding the inference that links them. What the combination produces is the interpretation, not the average: a rise of ten per cent is compatible both with people finding the system useful and with people being nagged into responding, and the interviews decide between those readings. Disconfirming evidence belongs in the display as a row of its own, with the case that does not fit and the explanation that failed to cover it. Where no integration point is named, the honest description is two studies in one report.

The phrase 'available and willing' describes convenience sampling, which Etikan, Musa and Alkassim (2016) allow only where randomisation is impossible, resources are limited and the study does not aim to generalise beyond the people studied. The student must first define the sample universe, stating who is included and excluded and why, which is the first of Robinson's (2014) four concerns, since a sample that cannot say what it excluded has not defined its universe. The student should then choose a strategy that answers the question and name the alternative it rules out. If the aim is to find shared patterns that survive difference, maximum variation sampling (Patton, 2015) selects staff across roles and shifts, and the rejected alternative is a typical case sample that would smooth over that variation. If the aim is to understand one setting deeply, a single critical case, defended through social representativeness and transferability (Gobo, 2004), is the better route. The student should also fix what the sample could have shown, in the spirit of Luborsky and Rubinstein's (1995) call for qualitative clarity as the analogue of statistical power. To show the sample can carry the conclusions, the student should state the frame, the intended sample, the sample actually obtained and any attrition, as Campbell and colleagues (2020) recommend, and report whether the analysis needed 16 or fewer interviews for common themes or up to 40 for cross-site metathemes (Hagaman and Wutich, 2017).

The sentence fails in three ways. It borrows a number from other studies, which Guest, Bunce and Johnson (2006) and Hennink and Kaiser (2022) do not license, since their ranges came from relatively homogeneous populations with narrowly defined objectives and are explicitly bounded to those conditions. It names no analytic target, although Hennink, Kaiser and Marconi (2017) found code saturation at nine interviews and meaning saturation only at sixteen to twenty-four in one dataset, so fifteen is uninterpretable until the student says what is being saturated. It asserts saturation rather than demonstrating it, which is what Vasileiou and colleagues (2018) found across 214 studies, where sample size justifications were limited and unrelated to the number of interviews conducted. A defensible rewrite states the aim in one sentence, names the target (for example, a richly textured understanding of a theme rather than a full codebook), and replaces the expected number with a procedure. If the analysis is reflexive, the student should follow Braun and Clarke (2021) and argue the stopping decision in interpretive terms, since meaning is generated rather than exhausted. A middle path exists: the student can cite Malterud, Siersma and Guassora (2016) on information power and judge the sample against the five dimensions, or adopt the reporting method of Guest, Namey and Chen (2020), publishing a base size, a run length and a new information threshold so a reader can see the point at which collection stopped. The counter-argument to answer is that the empirical evidence converges and can be modelled: Hennink and Kaiser (2022) report a consistent range, Weller and colleagues (2018) attach a median of 75 interviews and a range of 15 to 194 to a bounded domain, and Tran and colleagues (2017) predict theme accumulation with a mean error of 3 to 1 per cent. The reply is that those results answer questions about completeness and salience of a bounded domain, not about the depth of a meaning, so they support the rewritten proposal rather than the original sentence.

The sentence treats the site as found rather than made, and fixes the yield before the study has begun. Blomberg and Karasti (2013) show that a field site is a construct, constituted by bounding, selection and choice, so a proposal has to say why this centre, these shifts and these activities were chosen and what was excluded. Six months buys something a single visit cannot, as Nardi's (2010) three years of participatory research in World of Warcraft shows, but duration is not the design. It creates obligations that the proposal must carry. Blomberg and Karasti record that access is a continuing negotiation, that the researcher's presence belongs in the data, and that consent gained early does not cover later material; Boellstorff and colleagues (2012) treat consent, review approval and the handling of logs and images as part of the design rather than an appendix. The stated deliverable also borrows a standard the literature disputes. Dourish (2006) argues that implications for design may fail to capture what an ethnographic study is worth, and Button and Dourish (1996) hold that an analytic description of how members organise a setting cannot be converted into prescriptions without ceasing to describe that setting. A defensible rewrite names the construction of the field, the analytic question, the access and consent arrangements, and the specific claims the study will support. The counter-argument to answer comes from Kozinets (2002), who presents netnography as faster and less expensive than traditional ethnography, and from Seaver (2017), who offers tactics that work from an outsider's partial position without access. The reply is that those accounts relocate the obligations rather than remove them: the researcher still has to show how the site was bounded, what was observed and who agreed, and a design project still has to produce the warrant for its own claims.

The claim is not yet a video finding because its unit is the person rather than the sequence. Heath, Hindmarsh and Luff (2010) argue that the record is what makes gaze, gesture and turn timing available, and the unit of analysis becomes the ordered exchange in which one action makes the next possible. The team cannot report that the nurse looked confused, which is a claim about a mind; it can report that she withheld her next turn until the first speaker had finished and that the following question narrowed the topic, a claim about a record a reader can inspect. Before the camera rolled, the team should have defended the recording and framing decisions that fix what could later be claimed: where the camera was placed, what it framed, whether collection ran in one period or in phases, what would happen to the material, and how consent was obtained for events that had not yet happened (Heath, Hindmarsh and Luff, 2010). A single fixed frame has already judged gesture out of scope, and Pink (2013) makes the ethics of representing participants part of what the record means rather than a condition attached to it. A data session, in which colleagues repeatedly view the same clip and argue about what it shows, is the discipline that grounds a reading (Jordan and Henderson, 1995); a second analyst who reads the same thirty seconds differently is the method working, not a failure. The contested question then decides the final move. If the warrant is the participants' own displayed orientation, as on the sequential view (Mondada, 2013), the team must show the coordination the reader can see by slowing the clip, and Mondada (2016) frames multimodality as a different object of analysis rather than an added column. If instead the warrant lies in the analyst's composition, as Menning and Murris (2024) argue through filmmaking practices that shape what counts as data, the framing becomes part of the account and must be documented. What would settle it is a demonstration that separates the two grounds: independent analysts coding the same clips and reporting where their accounts diverge, and the raw and edited records published side by side so a reader can try to reconstruct the claim from the unedited material. If the claim survives that reconstruction, the warrant is in the participants' orientation; if it does not, the construction is doing the work.

The team's plan is not yet a discourse finding, because a theme count is the output of a different question. Taylor (2013) argues that language is constitutive rather than a neutral channel and that there is no single discourse analysis, so the label alone specifies nothing. On the discourse route, the object is what a version of events does: Edwards and Potter (1992) treat memory and attribution as things people do in talk, and Potter and Wetherell (1987) show how accounts are built to manage blame. Hodges, Kuper and Reeves (2008) sort the field into three clusters, formal linguistic, empirical and critical, and Gee (2014) offers seven building tasks, so the team must say which cluster and which task it is using. On the narrative route the object changes again: Labov and Waletzky (1967) ask how events are ordered into a six-part structure, and Riessman (2008) offers four kinds of narrative analysis, thematic, structural, dialogic and performance, and visual, while Chase (2005) shows that the chosen lens fixes what counts as evidence. The contested question is whether a claim may travel from the local order of talk to the cultural resource behind it. Wetherell (1998) answers the conversation-analytic reservation by combining interpretative repertoires and positioning, while Fairclough (2013) and Gee (2014) hold that a text is tied to power and ideology, so an analysis confined to the turn misses what the text enacts. What would settle it is one corpus analysed twice under each warrant and reported so that a reader can see which inferences survive; De Fina and Georgakopoulou (2011) add that narrative's model case is the small story in interaction, not only the elicited one. The team should name its object, its warrant and its point of contact before coding, as Alejandro and Zhao (2024) do when they combine four families of method.

The claim 'residents co-analysed the interview data' is a claim about authority and must be shown, not asserted. Arnstein (1969) graded participation by who decides, with informing and consultation in the tokenism group and partnership, delegated power and citizen control in the citizen-power group, so the team must say which rungs the residents occupied and for which decisions. Cornwall and Jewkes (1995) locate the difference between participatory and conventional research in the location of power, and ask who defines the problem, who analyses, and who owns and acts on the findings, which gives the team three questions to answer in the methods section. Participation also needs conditions: Bodker (1996) argues that it requires material and organisational support and that conflicts of interest are a resource for design, while Kensing and Blomberg (1998) treat the politics of design and the transfer of results to workers and user groups as live issues. The obligation is specific. Greenberg (2025) has participants interpret data rather than only generate it and requires the researcher to document which analytic decisions participants made, and Bratteteig and Wagner (2016) show what that documentation would look like by analysing participation as linked decisions, so a small choice opens or closes later ones. If the team adds photographs, the claim changes again: Wang and Burris (1997) show that photovoice moves the collection and first reading of images to participants and states three goals, and Eisner (1997) warns that form shapes understanding while risking a loss of evidential warrant, so criteria must fit the form. The unit's contested question sets a grading position, held by Arnstein (1969) and Cornwall and Jewkes (1995), against a paradigm position, held by Heron and Reason (1997), for whom participants are co-researchers within an extended epistemology, and by Bergold and Thomas (2012), who prefer justificatory arguments addressed to particular stakeholders over one scale. The grading side would ask for a decision log; the paradigm side would ask for an argument that shows residents' knowledge shaped the findings. A comparison of projects reporting both a decision log and participants' accounts of what their involvement was worth would settle it, and Barab and Squire (2004) show why leaving the warrant open is costly.

The team's first task is to say what kind of design they have and where its strands meet, because content is not a design. Fetters, Curry and Creswell (2013) distinguish integration at the level of design, where the three basic configurations are exploratory sequential, explanatory sequential and convergent, at the level of method, where strands connect, build, merge or embed, and at the level of interpretation and reporting, where they meet through narrative, data transformation or a joint display; a trial with interviews is usually explanatory sequential or convergent, and the team must pick one and defend it. Morse (1991) shows that location is part of the decision, since her notation records whether the strands run together or in sequence and which one leads. The title they propose also has to survive the field's own looseness: Johnson, Onwuegbuzie and Turner (2007) collected 19 definitions from leaders, most of which turn on combination alone, so a claim of mixed methods is easy to make and proves little; Plano Clark and Ivankova (2016) define integration as an explicit interrelating of the components, which is the stronger bar. The audit evidence is discouraging. O'Cathain, Murphy and Nicholl (2007) classified 18 per cent of 647 health services research studies as mixed methods and found the drivers pragmatic, including funding, and Zhang and Creswell (2013) reviewed 30 articles and found three mixing procedures, the first of which analysed the two data types separately and integrated them only at interpretation. O'Cathain, Murphy and Nicholl (2010) supply the remedy: a triangulation protocol that lists findings on one page and counts silence and dissonance as results, a mixed methods matrix that sets cases in rows against the data held on each case, and following a thread, with the post-analysis work timetabled as Morgan's third effort. For a convergent design, Moseholm and Fetters (2017) offer named analytic frameworks that merge data during analysis, and Guetterman, Fetters and Creswell (2015) show in 19 published joint displays what an integrated result looks like when the arrangement carries an inference neither strand produces alone. The contested question then applies to them directly. The design position, held by Fetters, Curry and Creswell (2013) and O'Cathain and colleagues, demands that they name the point of integration and show it in a display; the conceptual position of Bazeley (2024), who argues that integration is constituted by interdependence that is connected, transactional, transformative, coherent and dynamic, and of Fetters and Molina-Azorin (2017), who spread integration across philosophical, methodological and methods dimensions, says that no single display captures what their two strands do to each other over the life of the study. What would settle it is a reader test, in which reviewers who do not know the declared design judge whether the reported inference depends on both strands, and a subtraction test, in which one strand is removed and the inference is re-examined. Greene, Caracelli and Graham (1989) already showed why the tests matter: they found evaluations that declared the purpose of triangulation without a design able to deliver it, which is the failure the team can avoid by naming the join before they write.

The candidate's methods section is doing two different jobs badly. The appeal to pragmatism is a warrant that has to be argued rather than invoked: Biesta (2010) separates stronger and weaker versions of the claim that pragmatism can ground mixed methods research, identifies seven levels at which ideas about mixing occur, and concludes that Dewey's pragmatism dissolves certain dualisms rather than settling the paradigm question. The candidate therefore owes an account of why a survey and an interview schedule are commensurable here, and of what each is taken to be measuring. The paradigm dispute is not settled, and three positions bear on their design. Howe (1988) argues from a pragmatic perspective that no incompatibility exists at the level of practice or of epistemology, which would licence their plan if it is right. Sale, Lohfeld and Brazil (2002) argue that the two paradigms hold different views of reality and therefore study different phenomena, so the methods cannot be combined for cross-validation or triangulation and may be combined only for complementary purposes. Triangulation is precisely the purpose the candidate has named, so this position, not the caricature of purism, is the one their design has to answer. Giddings (2006) adds the reason to be careful about instruments, arguing that the promise of the best of both worlds takes on a certainty that serves economic restraint and evidence-based practice, and that mixed methods covers for a continuing positivism. On appraisal, the candidate may claim that a tool exists and can be applied: Hong and colleagues (2018) present version 2018 of the Mixed Methods Appraisal Tool, with five core criteria for each of five design categories, and Pace and colleagues (2012) found that two reviewers appraising 32 participatory research evaluations with the pilot version agreed from moderate to almost perfect on its criteria at about 14 minutes per study, while concluding that development was still needed. What the candidate may not claim is that quality is thereby settled. O'Cathain (2010) states that no accepted criteria for judging a mixed methods study exist, and Fabregues and colleagues (2024) argue that frameworks check whether integration is present rather than whether it was rigorous. The reporting burden follows: Fabregues and colleagues (2020) screened eight palliative care journals, included 159 mixed methods studies, and found that authors usually did not describe the design type and gave little detail on integration, which is the failure the candidate must avoid. To let a reader judge the meta-inference, the report has to name the point of integration as a design decision, following Schoonenboom and Johnson (2017), who set out seven design dimensions including point of integration and keep validity in view throughout, and it has to show how the interview findings change what the survey count means, as Creamer (2018) requires when she makes integration constitutive of the method. Finally, the candidate should state what would settle whether the conclusion is mixed: reviewers who did not run the study would rate the report with a named instrument and then judge, blind to those ratings, whether the conclusion depends on both strands; an inference that survives the removal of either strand was mixed, and one that does not was two studies in one report.

The reviewer's inference runs in the wrong direction. Tong, Sainsbury and Craig (2007) built COREQ by compiling 76 items from 22 existing checklists and covering the research team and reflexivity, study design, and data analysis and reporting, but the method consolidates prior opinion rather than testing what predicts quality, so a full score shows that a report is complete rather than that its analysis is sound. Barbour (2001) states the objection: a checklist can be completed without the analytic work it is meant to evidence, so procedure can substitute for judgement. O'Brien and colleagues (2014) make the same limit explicit for their 21-item SRQR, whose purpose is transparency, and transparency is a property of a document, not of an analysis. The editor should also see that the debate is not settled. On one position reporting criteria improve quality because transparency is a precondition for appraisal (Tong, Sainsbury and Craig, 2007; O'Brien and colleagues, 2014); on a second they measure reporting rather than quality (Barbour, 2001; Tracy, 2010); on a third the criteria are misconceived, because traditions do not share the assumptions a single set would need (Rolfe, 2006; Sandelowski, 1993). A fourth, sharper view holds that transparency and reflexivity are themselves a writerly performance, so a report can display trustworthiness without possessing it (Van Cleave and Marn, 2026). The policy would be justified by evidence, not by assertion: rate a sample of reports with COREQ, and have reviewers blind to those ratings judge whether each study's interpretation is warranted by its evidence. If the ratings predict that judgement the instrument reaches quality; if they predict completeness only, the policy measures the report. The reviewer should therefore judge what no COREQ item asks, which is whether the interpretation is warranted, whether the study states which kind of generalisability it claims (Smith, 2018), and whether the approach is methodologically coherent with the tradition it borrows from (Wilson, 2025).

The 300-word statement is the confessional move the unit sets against a documentary standard. It is written after the analysis, so it narrates the researcher's position rather than exposing the decisions that produced the reading, and Mauthner and Doucet (2003) warn that such a retrospective account can tidy messy interpretive work into a story with no decisions in it. The supervisor should ask for the decision record instead: which codes were tried, which extract resisted the reading, and what evidence would defeat the interpretation, communicated as part of the method rather than added at the front, which is the form Olmos-Vega and colleagues (2023) recommend. The dispute over whether this record is evidence runs three ways. On the documentary position the record exposes the decisions and can be assessed (Finlay, 2002; Olmos-Vega and colleagues, 2023; Probst, 2015). On the performative position reflexivity is a methodological power that reassures the reader without disturbing authority (Pillow, 2003). On the concealing position the record can be present and still not be evidence (Mauthner and Doucet, 2003). The evidence that would settle it is a blind comparison: two versions of one study, with and without the record, given to reviewers asked whether the interpretation is warranted and whether a second analyst could reconstruct the coding decisions. If the record moves the judgement of warrant, it is evidence; if it moves only a rating of thoroughness, Pillow's reading holds. The student should also note that writing is itself a method of inquiry (Richardson, 2000) and that the account's form carries a claim (Van Maanen, 2011).

A strong answer separates the three questions the brief asks, because conflating them is the usual failure. What is adapted: the order and composition of a reading list shown on a course page, or the notifications sent about new items. The signal should be named precisely, and the honest design uses explicit signals, such as topics a student selects and items they save, alongside implicit ones, since Jannach and colleagues describe both routes to a user model and note that a click is not an endorsement. Give the cold start a design answer rather than hoping it away: for a student with no history, offer a subject-level list that says it is not yet personalised, and ask two or three questions that a first-year can answer. Then name the loop. If the recommender only surfaces items similar to what a student has already borrowed, the library's holdings stop being discovery and its click data records its own suggestions, so the system collects evidence for a narrowing view of the student. The evidence that would show harm is specific and modest: a rising concentration of loans within one subject class for students whose recommendations are personalised, measured against a matched group on a non-personalised list, plus interviews asking whether students found anything they would not have found otherwise. Track the dismissal rate of recommendations, and treat a falling rate of first-time borrowings from outside the student's main subject as a signal. The negative finding has to be stated too: if the personalised group borrows more overall and no less widely, the loop is not yet doing damage, and the honest report says so rather than asserting either case.

The design should be argued from what each method can and cannot establish. Model-based judging is usable and cheap, and Zheng and colleagues show a strong judge can agree with human preferences at roughly the level humans agree with each other, so a blanket refusal is not required. The circularity has to be handled rather than acknowledged. Do not let the judge come from the same family as the system under test, and do not let it be a model tuned on the same preference data, because Sharma and colleagues show preference data rewarding convincingly written sycophancy over correct answers. Keep a stratified human sample, roughly one task per category, to calibrate the judge, and report agreement between judge and humans rather than only the judge's scores. Use the judge for breadth and humans for validity. Red teaming then covers what a benchmark cannot reach. Ganguli and colleagues publish their instructions and statistical methods and are candid about their uncertainty, and the limits transfer: red teaming samples attacks, so it cannot estimate how often harm would occur, and it finds what the team thought to look for, so an absence of findings is not a clean bill of health. Report the version and date of the model tested, since the artefact will be updated after the report is written, and say which failures your method could not detect. The report should name the raters and how they were recruited, the agreement between them, what was held out from the judge's view, and the cases where the feature should refuse rather than answer. A reader who cannot see those details cannot judge whether the result survives release.

The held-out accuracy does not answer the complaint, and the unit supplies the reason. Accuracy is measured on data the system influenced and cannot establish that the inference is right about this person, which is the distinction between an accurate model and an accountable one (Fischer, 2001; Zukerman and Albrecht, 2001). The complaint is about a specific inference, and a held-out score is silent on it. The design could follow the scrutability position: show what the system has inferred about the person and why, allow the person to correct it, and allow them to refuse the adaptation (Kay, 2006; Bull and Kay, 2010). This is an architectural choice rather than a display one, and systems such as Personis and PebbleBed have been built on that basis, with their failed principles named (Kay and Kummerfeld, 2012). The evidence for shared control is real but narrow: a mixed-initiative system that showed its reasoning and let the user decide was preferred to a purely adaptable alternative and improved task performance (Bunt, Conati and McGrenere, 2007). The counter-position has to be answered too: attending to an adaptation carries a cost (Jameson, 2007), and the control that exists in the field is often cosmetic (Jannach, Naveed and Jugovac, 2017). What would settle the question is a controlled study of model visibility that measures recovery from a model error, with the model's state checked before and after correction, plus deployment evidence that correction changes behaviour rather than only the display.

The engineer's argument is not supported by the evidence this unit carries. Held-out accuracy is a property of the model, whereas the adaptive hypermedia field had argued that an interface which changes unpredictably, obscurely or uncontrollably is a design failure, and had set the requirement that users be able to inspect the adaptivity mechanisms and control their results (Hook et al., 1996). INSPIRE implemented that requirement as a learner able to move from full system control to full learner control and to intervene in lesson generation (Papanikolaou et al., 2003), which is a design precedent the engineer would be discarding. The colleague is right that the question predates recommendation, but the further claim that the field settled it is too strong. The empirical record on link adaptation is conditional rather than settled: the founding benefit came from informal studies that aimed to identify when such adaptation is useful (Kaplan, Fenwick and Chen, 1993); the joined educational system reported different effects on different types of users rather than a main effect (Weber and Specht, 1997); and twelve years in, no generic model could be stated because systems were bound to single applications (Knutov, De Bra and Pechenizkiy, 2009). To settle whether the older design rule was dropped for a good reason, the team would run a comparison that holds the authoring budget constant, adapts links at different strengths against a stable baseline, measures an outcome beyond short-term achievement, and reports the quality of the model driving the adaptation. Only under those conditions could a null result be attributed to the technique rather than to the model, the authoring or the measure that concealed it.

The engineer is right about the engine and wrong to conclude that no reason can be given. Matrix factorisation represents people and items as latent vectors, and Koren, Bell and Volinsky (2009) present it as superior to nearest-neighbour methods precisely because those factors absorb implicit feedback and temporal effects, which means the numbers carry no name and an interface has nothing to print until a labelling step is supplied. The product manager's sentence therefore describes the item, not the computation, so in one sense it is a fiction. The explanation literature, however, shows that this is not automatically disqualifying. Herlocker, Konstan and Riedl (2000) found that the components of an explanation differed in how compelling they were and that explanations could improve acceptance, which places the effective variable in the presentation. Tintarev and Masthoff (2010) require an evaluation to state which aim it is measuring, because transparency and persuasion do not move together, and Millecamp and colleagues (2019) found the benefit to depend on the person, helping those with a low need for cognition while creating a lack of confidence in those with a high need for cognition. Against the objection, McAuley and Leskovec (2013) show that the gap can be closed rather than papered over, since combining ratings with review text produced interpretable textual labels for latent rating dimensions and improved prediction. To decide, the team would run two comparisons: one holding the engine fixed and varying only the basis of the reason, and one holding the reason fixed while varying the engine, measuring in both whether the person can check the reason against the item and whether the check changes the choice. If the item-attribute sentence cannot be checked, the team is measuring persuasion, not justification, and should say so.

The popularity plan assumes a population default: with no evidence the system shows the median listener, and the person who is not that median listener spends their first sessions correcting an error built in before they arrived. Hu, Koren and Volinsky (2008) show how strongly this logic can bite, because their implicit-feedback model treats the absence of an event as weak evidence of disinterest, so silence is read as a signal. Lam and colleagues (2008) and Saveski and Mantrach (2014) give the popularity plan its best case, showing that a population-level estimate or an item's own properties can produce substantial and consistent improvement over pure collaborative filtering, and Cremonesi and colleagues (2011) shrink the cold start further by importing a profile from another domain. The two-question plan assumes the person will state a preference cheaply and accurately. Rashid and colleagues (2002) support this, since the choice of which items to put to a new user significantly affected both the effort spent and the accuracy that followed, in a live study of over 300 users, and Christakopoulou and colleagues (2016) report a 25 per cent improvement over a static model after only two questions. What the team cannot know is how far those results transfer: the elicitation studies are prototypes on selected data, Jannach and colleagues (2021) find conversational recommenders evaluated inconsistently and rarely on the conversation itself, and Rashid and colleagues (2002) measured the effort those questions cost the person answering them, which is the cost side of asking that the unit keeps on the balance. The team should therefore run a randomised field comparison: hold the recommender engine fixed, assign new accounts at random to the popularity default, to a two-question elicitation, or to an imputed prior, and measure the value of the first session, the effort the person spent answering, and the diversity of what they consumed over the following weeks. Accuracy against logged data is not enough because the logged data were produced under an earlier interface, so the team needs the individual-level diversity measure Nguyen and colleagues (2014) built, and it needs the first-session value that no study in this list reports.

An offline figure supports a comparison inside one protocol: it can show that one model agrees more often than another with choices recorded under the old system. It cannot show benefit, because the log records what the old system showed and what people then did, so a model fitted and scored on it is rewarded for reproducing past exposure. Several declared-or-undeclared choices could produce a 12 per cent gain with no user benefit. The split of interactions into training and test data is one, since Meng and colleagues (2020) found the splitting strategy an important confounding variable that can markedly alter the ranking of models across three strategies, seven models and two datasets. The metric is another: Krichene and Rendle (2020) show that sampled metrics are inconsistent with their exact versions and do not preserve relative statements even in expectation, and Herlocker and colleagues (2004) show that accuracy metrics collapse into roughly three equivalence classes that do not correlate across classes, so a metric change can move the comparison. Baseline choice and tuning are a third, because the audits of Ferrari Dacrema and colleagues (2019, 2021) found that most neural methods could be outperformed by simple alternatives under consistent baselines. The retailer should therefore require a pre-registered field comparison: hold the deployed system fixed, randomise users between it and the candidate, and report the offline-online correlation per design choice. The measures should be interventional or business-level, following Jannach and Jugovac (2019), who find that field tests rarely report the business measures an organisation cares about, rather than the offline number that is being offered as the evidence.

The click metric tells the platform how one party, the listener, responded to what was shown. It cannot tell the platform whether providers received a fair share of exposure, because the objective as written rewards popular items; Abdollahpouri, Burke and Mobasher (2017) show that learning-to-rank algorithms amplify popularity bias and that a regularisation framework can trade accuracy against long-tail coverage at little cost to ranking quality. It also cannot tell the platform whether listeners were served in the longer run, since engagement and satisfaction diverge: Zhao and colleagues (2018) found in a field experiment with 1,508 users that an implicit-optimised model engaged people more while also raising negative engagement correlated with lower satisfaction. And the aggregate hides the distribution: Ekstrand and colleagues (2018) found popularity bias disadvantaging particular demographic groups in a way invisible in aggregate accuracy. A defensible metric set therefore has to name its stakeholders, which Sonboli and colleagues (2022) describe as consumers, providers and the platform, and to accept that satisfying one side's criterion can worsen another's. Because any single scalar encodes one party's preference, and Kleinberg, Mullainathan and Raghavan (2016) prove that three fairness conditions cannot hold simultaneously except in constrained cases, the platform must state which interest the default serves rather than pretending the choice is technical. I would require the metrics to be reported together: ranking quality, long-tail coverage and provider exposure by label size, listener satisfaction measured subjectively rather than by clicks, and a consumption-diversity measure following Vargas and Castells (2011). For the study design I would require a randomised field experiment holding the current system fixed, with the re-ranking intervention applied to a random subset of listeners, reporting the cost of the intervention against each objective, as Beutel and colleagues (2019) did when they improved pairwise fairness in a production recommender. I would also ask the platform to record the long-run effect rather than the first-session effect, because Chaney, Stewart and Engelhardt (2018) show that training on confounded data homogenises behaviour without raising utility, and Jiang and colleagues (2019) show how the loop tightens. Finally I would report the design as a values decision in the sense of Stray and colleagues (2024), and note with Deldjoo and colleagues (2023) that the absence of shared benchmarks means the platform cannot borrow a settled standard and must state its own.

The logs measure two things and conflate them. What each user was shown is exposure, which the ranking function sets, and what each user opened is selection, which the user sets. Bakshy, Messing and Adamic (2015) measured both on one platform and found that individual choices limited cross-cutting exposure more than algorithmic ranking did, across 10.1 million users, so a parliamentary claim built on opened articles alone cannot support a conclusion about the algorithm. The logs can establish an exposure distribution, which is an auditable artefact, and they can show whether that distribution narrowed over time; they cannot show what anyone believes, and they cannot separate a narrowing caused by the ranking from one caused by the users it was serving. Moller and colleagues (2018) compared recommendation logics against human editorial selection on a Dutch broadsheet and found the logics produced sets about as diverse as editors, which is why the audit has to include the counterfactual rather than the deployed system alone. The defensible claim from these logs is that the platform controls and can report the exposure it produced. Moving from exposure to belief requires the design Guess and colleagues (2023a) obtained: platform-side randomisation of the ranking for consenting users, with attitudes measured before and after, because the same experiment found large changes in exposure and time spent and no significant change in issue polarisation, affective polarisation or political knowledge over three months. I would require that experiment, run for longer than three months and paired with the companion design of Guess and colleagues (2023b) so that reach and persuasion are separated, and set alongside a measure of network structure, since Tornberg (2022) locates the harm in partisan sorting rather than in narrowing. Before the intervention I would report the baseline exposure distribution and the current selection pattern, note that echo chamber membership is confined to a small segment of the population in survey work (Dubois and Blank, 2018) and that online audiences are more polarised than offline ones in most countries but not all (Fletcher, Cornia and Nielsen, 2019), and treat selective exposure as a mechanism that predates the feed (Stroud, 2010). The outcome would be a stated effect on attitudes or an honest null, and the conclusion would follow the measurement rather than the accusation.

The proposal bundles two propositions. Promoting frequently used functions is a claim about the level of automation applied to the task of finding them; reordering the toolbar is a change to the surface that administrators have learned by habit. The evidence supports adapting some things and not others. Findlater and McGrenere (2004) found that a static split menu was significantly faster than an adaptive one in a controlled study with 27 participants, and that the adaptable menu, rearranged by the user, was preferred by most participants. Gajos, Czerwinski, Tan and Weld (2006) compared three adaptive interfaces with a non-adaptive baseline and found that the spatially stable Split Interface improved performance and satisfaction while their Moving Interface, which promoted only hard-to-reach functions and so carried the higher theoretical benefit, did not improve either. The reason is that movement invalidates spatial memory, and movement is not the only design choice: the same work found that adaptation performed once per session produced positive results while adaptation available at every interaction fared worse than a fixed interface, and that hiding features reduces awareness of them (Findlater and McGrenere, 2007). I would therefore refuse the reordering toolbar and approve an advisory design instead. The system may recommend which functions to promote, and the administrator decides, which is the mixed-initiative position Bunt, Conati and McGrenere (2009) argue resolves the dilemma between an adaptable interface that burdens the user and an adaptive one that surprises them, and which the field evaluation of McGrenere, Baecker and Booth (2007) found gave better navigation and learnability than automatic adaptive menus. Before approving even that, I would require a field trial rather than a laboratory study, with expertise measured, findability, awareness and task time recorded separately, and replication at a second site, because the strongest studies in this literature are small and mostly predate 2010.

The vendor's claim is about language, and the unit's evidence says the failures concentrate elsewhere. Fluency is not task success: Gao, Galley and Li (2019) show that measured progress and reliable evaluation concentrate in dialogue tasks with a checkable answer, and that automated surface metrics correlate weakly with human judgement, so a fluent interface may still fail its user. Luger and Sellen (2016) found users could not judge what the agent had understood and needed visual confirmation on all but the simplest tasks, a gulf of evaluation that fluency does not close. I would require two bodies of evidence. First, a deployment in the bank's own channel, not a laboratory study, reporting task completion on card blocking and payment queries separately from satisfaction, because Porcheron and colleagues (2018) and Beneteau and colleagues (2019) show repair effort is often shifted onto the user or to bystanders and is invisible in an aggregate success rate. Second, a repair comparison: the same recognition accuracy with and without options, explanations and a route to a person, since Ashktorab and colleagues (2019) found offering options most favoured and deferring to a person preferred once repair failed. For design, I would require that the assistant say what it understood, offer options after a misunderstanding, and hand difficult queries to a human by design, as Grudin and Jacques (2019) describe for human-bot teams. Because Chaves and Gerosa (2020) found the chatbot design literature largely exploratory, I would require these choices be tested in the bank's own setting rather than assumed to work.

The vendor's evidence mixes two different claims. Bloom (1984) reported about two standard deviations for one-to-one human tutoring with mastery learning, a claim about a person giving feedback and correction rather than about a program, and VanLehn (2011) found that even that figure does not survive pooling: human tutoring came out at about 0.79 standard deviations and step-based computer tutoring at about 0.76, so the machine approaches the human tutor and neither approaches two sigma. A single study reporting 0.8 standard deviations therefore cannot be an unusual result, and it cannot be used to claim the benchmark. The pooled evidence is more informative and more qualified. Kulik and Fletcher (2016) reported a median gain of 0.66 standard deviations over conventional instruction across 50 evaluations, with the size depending heavily on whether the outcome test was locally developed or standardised, and Ma and colleagues (2014) showed that the effect depends on the comparison, reporting about 0.42 against teacher-led large-group instruction and no significant difference against individual human tutoring. Steenbergen-Hu and Cooper (2013) found effects from about 0.01 to 0.09 in school mathematics, and their college analysis (2014) found later studies weaker than earlier ones. I would therefore require: the platform's own randomised comparison against the tutorials it replaces, with a standardised outcome measured after a delay rather than immediately; a statement of which outcome was used; and evidence on how many learners complete the sequence, since Reich and Ruiperez-Valiente (2019) show that scale alone does not deliver completion. For the design question, I would hold the practice material and the number of practice opportunities fixed and vary only whether item order is chosen by the system's tracked model or fixed in advance. Koedinger and colleagues (2023) found that learners need about seven practice opportunities per knowledge component while differing little in rate, so the plausible benefit lies in the number and timing of opportunities. If adaptive sequencing beats fixed sequencing with practice held constant, the model is doing work; if the conditions are indistinguishable, the platform is a scheduler, and Kalyuga's (2007) expertise reversal evidence would still justify adapting the level of support as learners develop.

The judgement is not 'how long are waits'. It is whether any trust sits above a threshold the board has already agreed, and whether the trust in question is drifting. That is a comparison against a reference on one scale, plus a trend, so I would draw a sorted dot plot of the thirty trusts against a common horizontal scale, with the expected range shaded as a band and a sparkline beside each trust that shows its own twelve months. Position on a common scale carries the ranking, and the band turns a bare value into a comparison. Two alternatives fail for stated reasons. A heat map of trust by month would use colour for magnitude, and colour sits low in the channel ranking, fails for readers with colour-vision deficiency and hides the fine differences that matter at the threshold. A dual-axis overlay of waits and admissions would let two independently chosen scales manufacture a relationship, so the apparent correlation would be an artefact of the two scale choices rather than a finding. Checking whether it worked is the hard part. I would compare the design against the current chart on two named tasks with readers from the board, and count time, error and wrong escalations. I would state that the tasks were mine, and that choosing them teaches the reader what to look for, so the study reports on that search and not on decision quality. That limit is the reason I would also run a short think-aloud with two board members on their own live question, and code the insights they reach.

The premise is wrong, because on this dashboard the colour is the encoding rather than decoration. The values are carried by colour alone, so the palette is the measurement instrument. The mechanism has two parts. First, a rainbow palette is not perceptually uniform in lightness. It climbs quickly through the middle of the range and flattens at the ends, so equal steps in the data do not produce equal steps in what the eye sees, and readers infer boundaries where the underlying rate changes smoothly. Second, a rainbow scale can place two values that differ sharply in the data at hues that are difficult to tell apart, which is the same failure that a red and green pair produces for readers with colour-vision deficiency. The remedy is a sequential or diverging perceptually uniform palette chosen from the sign of the data, with a legend that states the class breaks. Before accepting the councillor's position I would need evidence that the palette does no harm on the reader's actual question. That means a judgement task comparing the rainbow scale with a uniform alternative, measuring accuracy and time on the specific judgement residents have to make, and a check that both scales remain readable under a colour-vision-deficiency simulation. The honest position is that the mechanism is established while the size of the effect on real decisions is not.

Both parties treat the ranking as a rule about displays, and the unit's argument is that it is a rule about channels measured alone. Cleveland and McGill (1984) had readers judge the relative size of two encoded values and ordered elementary perceptual tasks by error, placing position and length at the top and area, angle and colour saturation at the bottom, and Heer and Bostock (2010) preserved that order with crowdworkers using the same error measure. Because the replication preserved the order while narrowing the gaps between channels, the result is ordinal: position beats length, but the ranking does not license the team's claim that area is good or the reviewer's claim that it is forbidden. Heer and Bostock's own area experiment is the decisive evidence against both readings: rectangular area matched circular area in accuracy, aspect ratio mattered, and one-to-one ratios produced the worst judgements, which they explain by readers using one-dimensional length as a proxy for area. The authors also state that display, lighting and viewing distance were uncontrolled and that visual variables interact, so an isolated channel does not predict a real display. The judgement settles it. If the reader must compare a total against a target, a bar read by length serves the task, which is what Simkin and Hastie (1987) and Hollands and Spence (1992) show when display type interacts with the task and change and proportion judgements draw on different strategies. If the reader must see the composition of the whole, the treemap may serve it, though Spence and Lewandowsky (1991) warn that the form's accuracy depends on the particular proportion judgement required. Zacks and Tversky (1999) add that the form changes what the reader takes the data to be about, and Shah and Freedman (2011) show that prior beliefs enter comprehension, so perception alone does not decide the outcome. What would settle whether the ranking governs the choice is a study that assigns channels within the same analytical task and measures the quality of the decision, rather than the accuracy of reading an isolated value, and that reports how channels combine; Franconeri and colleagues (2021) name that laboratory-to-decision gap as the open problem.

The unit's thesis is that formal grammars and constraint systems let a machine propose an encoding, and that every recommender is only as good as the perceptual rules it encodes. A recommender is a grammar plus a scoring rule. The grammar, from Wilkinson (1999) and in layered form from Wickham (2010), is what makes a chart a specification a machine can generate, and Polaris (Stolte, Tang and Hanrahan, 2002) shows the same idea as an interface in which a visual specification produces both a display and its queries. The scoring rule is where the two parties disagree, and neither has it quite right. The team's defence rests on agreement with human choices, and VizML (Hu and colleagues, 2019) does report consensus chart-type predictions comparable to human performance from one million dataset-visualisation pairs. But agreement with existing charts is conformity to past design decisions, not evidence that a chart suits the question asked, so the colleague is right to distrust it. The colleague's own position, that a design must be traced to evidence, is also too strong, because the perceptual evidence inside these systems is itself contested. Draco (Moritz and colleagues, 2019) learns its constraint weights directly from graphical perception experiments, which are experiments on isolated judgements in abstract displays, and Voyager (Wongsuphasawat and colleagues, 2016) shows the practical consequence of a statistical and perceptual scoring rule: it increased the coverage of data variables an analyst considered, which is breadth, while its authors state that tools must balance rapid exploration against targeted question answering. Data2Vis (Dibia and Demiralp, 2019) makes the gap plainest, because it was scored on language syntax validity and visualisation grammar validity, so a well-formed specification was rewarded and a fitting display was not. Munzner (2014) and Amar, Eagan and Stasko (2005) supply the position that the task can be encoded, since task abstraction precedes form and ten low-level tasks give a system a vocabulary. The honest verdict is to keep and watch the tool rather than to withdraw or trust it. What would settle the dispute is a study that holds the data fixed while varying the analytic question and asks whether the recommendation changes as the task requires, and an evaluation of a learned recommender on task-relevant quality rather than agreement with existing charts; neither study is in this unit's list.

The unit's thesis is that the field converged on perceptually uniform, colour-vision-safe encodings, but that the strongest recent evidence shows the condemned rainbow map is not uniformly bad, so the consensus is narrower than the advice it produced. The team's first decision is conceptual, because an ordered variable needs a ramp that varies in lightness. Ware (1988) found that sequences varying in lightness permitted accurate reading while those varying mainly in hue did not, and Brewer (1994) separated sequential, diverging and qualitative schemes and warned against hue for ordered data, so a rainbow-like scale is the wrong family for the team's quantity unless a specific reading justifies it. The reviewer and the third colleague are each half right. Liu and Heer (2018) found that single- and multi-hue perceptually uniform maps both supported accurate decoding when well designed, that single-hue maps erred more over small value ranges, and that perceptual space plus colour naming predicted reading only poorly, so the reviewer's family is well supported even though the perceptual model behind it is weak. Reda, Zaid and Nacenta (2023) found that colourful, categorising scales made readers more likely to discriminate global, distributional features while making localised features less noticeable, which is the genuine advantage the third colleague has in mind, and they also report conditions in which such maps are more harmful than useful. What settles the argument for a public site is accessibility. Nunez, Anderton and Renslow (2018) report that colour-vision deficiency affects more than 4 per cent of the population and produced cividis, which is perceptually uniform in hue and brightness and read nearly identically by deficient and typical viewers, so the rainbow's task benefit does not answer the reviewer's constraint, because a benefit on one judgement cannot be traded for a failure for a fixed share of the audience. Schloss and colleagues (2019) add that colour-to-meaning is contextual, since the dark-is-more bias competes with an opaque-is-more bias depending on background and apparent opacity, so the team must also match convention and background, not only the palette. The contested question is whether the consensus is narrower than the advice or right for narrower reasons, and it would be settled by a pre-registered task battery comparing ordered-reading accuracy and colour-vision safety for the same palettes across global and local judgements, and by studies using colour-vision-deficient participants rather than simulations. The defensible advice is the default, not the prohibition: use a perceptually uniform, accessible palette, and depart from it only for a named reading, for a bounded audience, with the trade-off stated in the caption.

The unit's thesis is that uncertainty can now be shown with encodings that beat error bars on reading tasks, yet the field's evaluations measure reading accuracy rather than decision quality, so it cannot yet say whether the better display produces the better decision. The first proposal should be treated as a design decision, not a default. Correll and Gleicher (2014) found that the encoding of mean and error significantly changes how readers make decisions, and recommended gradient plots and violin plots as better alternatives for inferential tasks than bar charts with error bars, which makes the choice a trade-off among inferential tasks rather than a single corrected form. The newer proposal has real support and real limits. Hullman, Resnick and Adar (2015) found that readers using hypothetical outcome plots made much more accurate judgements about plots of two and three quantities, while accuracy was similar for all three representations for most questions about a single quantity, and Kale, Nguyen, Kay and Hullman (2019) found that observers recovered the correct trend at a lower level of evidence with hypothetical outcome plots than with static aggregate displays. Which uncertainty is shown also matters. Hofman, Goldstein and Hullman (2020) found that readers shown 95 per cent confidence intervals overestimated a treatment effect and were willing to pay more for it than readers shown prediction intervals, so the ward's summary statistic must be chosen to match the decision, not the convention. The third colleague is right about the evidence base, and the field's own reviews say so. Hullman, Qiao, Correll, Kale and Kay (2019) applied a taxonomy of six evaluation decisions to 86 user studies and found practice concentrated on performance and satisfaction measures with a bias toward accuracy rather than decision quality, and Kinkeldey, MacEachren and Schiewe (2014) found that most geospatial uncertainty work develops depictions and only a small part evaluates them empirically. Kale, Kay and Hullman (2021) show why that gap matters: across eight designs the ones supporting the least biased effect size estimation were not the ones supporting the best decision making, because users switched strategies and satisficed with heuristics, so a display optimal in theory may not be most effective in practice. The contested question is whether showing uncertainty improves decisions or chiefly affects reading, and it would be settled by a decision-level trial with the managers themselves, which is the kind of study Heltne and colleagues (2023) found missing for clinicians, together with an experiment measuring judgement and decision in one design, as Kale, Kay and Hullman (2021) did. The team should therefore choose the encoding that matches the stated decision, and treat the claim that it improves the decision as a hypothesis to be tested in their own meeting rather than a settled finding.

The unit's thesis is that visual analytics rests on the claim that interaction is what makes scale manageable, and that the evidence for the claim is mostly design frameworks and case studies rather than controlled comparison of interaction techniques. The engineer's inference from the mantra needs correcting first. Shneiderman (1996) proposed overview first, zoom and filter, then details on demand, with a taxonomy of seven data types and seven tasks, and he presented the mantra as only a starting point, so it is an organising heuristic rather than a finding that more interaction is always better. The agenda documents make a large claim. Thomas and Cook (2005) defined visual analytics as the science of analytical reasoning facilitated by interactive visual interfaces, and Keim and colleagues (2008) defined it as the combination of automated analysis techniques with interactive visualisations for understanding, reasoning and decision making on very large and complex data sets. Neither document tests that claim; they state it as the field's purpose. The team should know that the vocabulary exists and that naming is not evidence. Heer and Shneiderman (2012) set out twelve interactive task types in three groups, data and view specification, view manipulation, and process and provenance, and Yi and colleagues (2007) proposed seven categories such as select, explore, reconfigure and filter, while reporting that interaction had received relatively little attention from the community and that existing frameworks covered low level operations rather than the benefits interaction provides. The clinician should also hear the design discipline against unfounded additions. Wang Baldonado, Woodruff and Kuchinsky (2000) gave eight guidelines resting on a workshop discussion and the authors' design experience, and their rule of parsimony warns that extra views cost context switching and attention, so six more linked views must be justified one by one. The strongest evidence for the engineer has to be reported with its limits. North and Shneiderman (2000) ran two user studies of a snap-together system and found that an overview and detail coordination improved performance by 30 to 80 per cent over detail only and uncoordinated interfaces on most tasks, but that is one coordination type in one system with data savvy users on constructed tasks, measured by speed rather than by analytic quality. A coordinated system has also worked in a demanding setting, since Stasko, Gorg and Liu (2008) entered the 2007 VAST contest with Jigsaw, won its university division, and report a professional analyst using it, which is evidence of use rather than of comparative effect. The analyst's claim that the question is closed is wrong, and the measurement work points to what is missing. Gotz and Zhou (2008) logged users' actions and identified a level they called Actions, building blocks for insight provenance; Pirolli and Card (2005) modelled analysis as a foraging loop and a sensemaking loop; and Sacha and colleagues (2014) offered a knowledge generation model with a computer side and a human side running exploration, verification and knowledge generation. These give the meeting a way to measure the analysis rather than count clicks. The contested question is whether interaction has been shown to make scale manageable, and it would be settled by a controlled comparison of named interaction techniques against a defined analytic outcome, and by a study of interaction in use over time with the process recorded. The team should add a view only when it names the judgement the view supports, and should record what the meeting does with it.

The unit's thesis is that narrative visualisation trades reader exploration for author control, and that the field has documented its genres and rhetoric without settling whether stories improve comprehension or chiefly increase persuasion. The journalist's claim needs the evidence named. Segel and Heer (2010) analysed 58 published narrative visualisations and derived seven genres, among them the magazine style and the annotated chart, plus three patterns for interactive stories, the martini glass, the interactive slideshow and the drill-down, and they described the design as a balance between author-driven narrative flow and reader-driven story discovery; their method was a design space analysis, so comprehension was characterised rather than measured. Kosara and Mackinlay (2013) argued that presentation, and specifically storytelling, is the next logical step for visualisation research, which is an agenda rather than a finding. The editor's reply also needs correcting, because the loss of exploration is not the whole story. Hullman and Diakopoulos (2011) show that narrative devices shape interpretation and can mislead, so a reader's sense of having understood may come from the framing rather than from the data, and Amini and colleagues (2015) show in 50 professionally designed data videos that the same rhetorical devices arrive from film. The authoring work makes the trade explicit: Satyanarayan and Heer (2014) built Ellipsis, in which a story is authored as a state-based scene structure with dynamic annotations and decoupled coordination across views, so the reader's freedom becomes a parameter the author sets, and Lee and colleagues (2015) add the process of finding insights, making a story and telling it. The newsroom should treat the accumulated design knowledge as craft, not evidence: Stolper and colleagues (2016) derived 20 techniques from 45 curated stories, and Bach and colleagues (2017, 2018) proposed data comics and then design patterns, each contribution a catalogue built by reading examples. Tong and colleagues (2018) leave interpretation among the unsolved problems. Only two studies test the trade directly. Wang and colleagues (2019) compared comics with infographics in two studies, one controlled and one in the wild, and report that participants largely preferred comics for enjoyment, focus and overall engagement and that comics improved understanding and recall of information in the stories, and Boy and colleagues (2015) found across three field experiments that an introductory story did not increase exploration. Together these results fit the thesis: memory and engagement improved, exploration did not, and neither study isolates accurate data reading from the story's hold on the reader. The contested question is whether a story improves comprehension or chiefly persuades. It would be settled by a study that holds the data constant, varies only the narrative framing, and measures data-reading accuracy, delayed retention and the reader's stated belief as three separate outcomes. The newsroom should keep both its story and its exploratory tools, label the story as an argument shaped by framing, and treat any claim that the story has been proven to improve comprehension as unsupported until that study exists.

The thesis is that most misleading charts come from unexamined defaults rather than deliberate fraud, that named distortions differ greatly in how much they mislead, and that the remedy is a design and audit discipline together with an obligation to answer for what a chart does after publication. The statistician and the officer are both partly right and both wrong, and the unit's evidence shows why. Tufte (1983) supplies a detection rule: the lie factor compares the size of the effect a design shows with the size of the effect in the data, so a truncated axis and a bubble area both fail the test by construction. That rule is about charts, not about readers, and the officer's claim that correct data settles the ethics is therefore too quick. Pandey and colleagues (2015) built control and distorted versions of the same charts and ran crowdsourced studies. All three exaggeration techniques were significant at p < 0.001 but differed in size, from r = 0.66 for a stretched aspect ratio to r = 0.37 for a truncated axis and r = 0.34 for area as quantity, and an inverted axis reversed 78.95 per cent of readers against 2.5 per cent in the control, so severity does not follow reputation. Newman and Scholl (2012) show the second failure mode: points inside a bar were judged likelier than equally distant points above it, across both axis directions and several populations, which means even an untruncated default chart misleads. Correll and Heer (2017) move from accidental to deliberate, treating the designer as an intermediary who can attack a reader and naming four categories of manoeuvre, and holding that fidelity to the data is not the same property as accurate perception. Correll (2019) turns that into obligations covering provenance, the framing of results, dual use and the treatment of participants, which is the instrument the officer is missing: the agency must be able to answer for the framing, not only for the numbers. McNutt and colleagues (2020) name mirages, apparent findings produced by the data, the specification or the analyst's assumptions, and test for them by metamorphic testing, perturbing the data and the specification to see whether the conclusion survives a change that should not matter. The title is a separate problem again. Borkin and colleagues (2016) found that readers fixate on titles more than any other element and recall them most often, and Kong and colleagues (2018) found that title slant changed the perceived main message of the same chart, with 83 per cent of readers reporting the visualisation unbiased while half of the coded recalled messages were slanted. The contested question has three sides. Szafir (2018) and the intent-free definition of Pandey and colleagues (2015) treat the failure as a design default; Correll and Heer (2017) and Correll (2019) treat intent and consequence as the unit of analysis; Cairo (2019) and Bergstrom and West (2021) treat the remedy as reader literacy. The study that would settle it would rate published charts by expert judgement and by measured reader effect, apply each intervention, and measure a delayed outcome by decision rather than by self-report of bias. The advice to the agency follows: rebuild the axis and the area encoding, drop the causal title, and record the audit, because the chart can be honest and still produce a belief the data does not support.

The thesis is that dashboards succeed or fail on the decision they serve and on the definitions behind each measure, and that the research base is thin relative to their reach, so most dashboard guidance remains craft rather than evidence. The board's premise, that deploying a dashboard improves decisions, is therefore a hypothesis, not a finding, and the team should say so. Few (2006) supplies the craft the thesis contrasts with evidence: a dashboard is read at a glance, so it should follow the reader's decisions, and the team can use that to argue that sixteen gauges give no priority to any measure. Sarikaya and colleagues (2019) supply the missing definition, showing that the term covers several distinct artefacts from monitoring displays to analytic workspaces, so the team must state which one it is buying. Shneiderman and Plaisant (2006) argue that controlled laboratory tasks miss what an analytics tool is for, and Saraiya and colleagues (2006) ran a longitudinal study of a bioinformatics analysis from raw data to the insights sought, reporting that short studies miss long-term use; the board's claim cannot be tested in a one-hour task study. Yigitbasioglu and Velcu (2012) reviewed the multidisciplinary dashboard literature and found that little was known about the extent of dashboards' effectiveness, which is exactly what the team should report instead of a confident prediction. The definition layer is where the measures are made: Kandel and colleagues (2012) argue that what counts as a data error is context-dependent and needs human judgement, so each of the sixteen measures has a definition behind it that someone chose, and Battle and Heer (2019) show that the published record of analysis is thinner than the practice, so nobody has recorded how the numbers will be read. Dimara and colleagues (2022) interviewed and surveyed decision makers and found they need trade-off overviews, scenarios and interrogation tools, not a wall of gauges, and Oral and colleagues (2023) reviewed eighty-eight decision-focused tools and found most support visibility of alternatives and criteria but not the design and choice stages; the trust's dashboard is likely to sit at the information stage. Vartak and colleagues (2015) and Yalcin and colleagues (2018) show that good design can be automated, with a recommender that judges what is interesting by deviation and a system that lets novices build without chart-design instruction, but this moves the definition into a rule rather than removing it. The contested question is whether dashboard failure is a design problem or a decision problem, with a third position that the field's instruments cannot see the decision. The team should propose the study that would settle it: a corpus of deployed dashboards rated for design quality, each classified by the decision it claims to serve, with the decision actually taken measured over time.

The thesis is that the shift from task accuracy to insight and decision quality is right and incomplete, because the outcome measures are young, comparisons are seldom repeated, and the statistics overstate what one study shows. The report should therefore separate three things: what the study measured, what the design can support, and what the conclusion claims. North (2006) supplies the target, arguing that insight is the purpose of visualisation and should be measured, and setting out why it is hard: insight is subjective and its coding needs expertise. Saraiya and colleagues (2005) ran the study the field needed and found a warning inside it: insight counts and domain value did not depend on the participant's background, but believed learning did, with novices overconfident and software developers sceptical. Two outcomes diverged in one study, so a counted-insight difference alone does not license a claim about insight as analysts experience it. The design layer comes next. Plaisant (2004) named the obstacles, including tasks that are hard to define and the long time real use requires; Lam and colleagues (2012) hold that the scenario should determine the method and the measures, so the report must say which of the seven scenarios it is in; and Carpendale (2008) makes the trade-off explicit, since a study buys generalisability, precision or realism, and a thirty-participant laboratory session with three datasets is precise and not realistic. Stasko and colleagues (2000) show what such a comparison can do, ranking two space-filling tools on search and analysis tasks, with Sunburst generally favoured on first use while Treemap performance improved with practice, and what it cannot do, since neither tool was shown to improve anyone's work. The measurement literature sets the same limit. Behrisch and colleagues (2018) report a corpus of quality metrics organised into a common vocabulary because the community's intuitions were not comparable, and Micallef and colleagues (2017) show a perceptual model matching hand-made presets on exactly three calibrated tasks, which is the reach a metric can claim. The insight measure itself has a known contamination. Zgraggen and colleagues (2018) found that over sixty per cent of user-reported insights on synthetic data with known ground truth were false, because more views mean more comparisons, and Hullman and Gelman (2021) argue that interactive analysis is a form of statistical inference that needs a theory of graphical inference, without which interface objectives become contradictory. Kosara and Haroz (2018) then apply to this paper directly: replications are rare, and their six threats to validity include statistical flukes and overgeneralised conclusions, which is what a p = .03 with no replication and a general claim about insight amounts to. Dragicevic (2016) supplies the reporting standard, an estimation approach with effect sizes and interval estimates, and warns that a null result should not be worded as evidence of no effect; the same logic forbids turning a point difference into a general claim. The contested question matters for the report's tone. The thin-evidence position, held by Kosara and Haroz (2018), Zgraggen and colleagues (2018) and Dragicevic (2016), asks the study to be revised and repeated; the wrong-standard position, held by Greenberg and Buxton (2008), Isenberg and colleagues (2008), Carpendale (2008) and Lam and colleagues (2012), asks instead whether a counted-insight difference is the right outcome for a coordinated-view tool at all. What would settle it is an audit coding published comparisons for effect sizes and intervals, preregistered replications of canonical comparisons, and a demonstration that one claim can be measured under two scenarios. The practical recommendation is major revision: report intervals, name the scenario and the task, drop the general claim about insight, and add a grounded or longitudinal component that observes analysts on their own data.

The thesis is that once the outcome changed from perceptual accuracy to belief and decision, the effects a chart can show became smaller, more conditional and more dependent on the reader than the earlier optimism assumed. The agency's claim that a well-designed chart speaks for itself is therefore the claim to test. Franconeri and colleagues (2021) hold that visualisations communicate only when designers tailor them to their audience, so audience becomes a design variable, not an afterthought. Galesic and Garcia-Retamero (2011) show that audience is heterogeneous: their graph-literacy scale, validated on representative samples of 495 Germans and 492 Americans averaging about nine of thirteen items, found roughly a third of each sample low in both graph literacy and numeracy. Okan and colleagues (2011) found that icon arrays improved comprehension and confidence more for high-literacy readers, so the same aid can help the least literate least. Xiong and colleagues (2020) explain why untested design fails: primed participants predicted naive viewers would find the same pattern salient, so the designer's own reading is not a guide. Bylinskii and colleagues (2017) show the remedy, models predicting element importance that beat saliency and were validated with crowdworkers, so the agency could measure where attention lands. Garcia-Retamero and Cokely (2017) give the largest decision-level base, thirty-six publications and 27,885 participants, and find that transparent visual aids robustly improve risk understanding, which supports testing rather than dismissing charts. Lipkus (2007) names what is still missing: few randomised trials comparing formats on the same outcomes and little theory behind format choice. Haroz and colleagues (2015) found pictographs encoding data cost no reading speed and mildly aided long-term memory and engagement while superfluous images hurt retention, and Skau and Kosara (2016) found the central angle least important in pie and donut charts and donuts as accurate as pies, so the form's effect is small and specific. The contested question is whether the effect on belief and decision is established, held by Garcia-Retamero and Cokely, Okan and colleagues, Sanchez-Holgado and colleagues (2023) and Franconeri and colleagues, or smaller and conditional and measured on the wrong outcome, held by Lipkus, Hullman and colleagues (2011), Xiong and colleagues, Haroz and colleagues and Skau and Kosara. The evidence that would settle it is a randomised trial in the decision context itself, with the decision as the outcome, graph literacy entered as a measured moderator, and the design fixed in advance, repeated across populations and formats. The agency should therefore test its chart on a sample that includes low-literacy readers, measure the decision rather than the impression, and report the literacy distribution beside the average.

A strong answer names decisions and gives a reason for each, and it holds one system throughout. Recruit in the pharmacies, with counter staff introducing the study, because a convenience sample of students would not test the medication problem the app exists for. For consent at a distance, use a short document written to be read at home, give a named contact, and make withdrawal possible without a stated reason; because reminders can reveal a health condition to a partner or a child, decide what appears on a locked screen and offer a discreet notification. On hardware, participants use their own phones, so publish the supported versions, test each phone at setup, and provide a one-page card plus a call in the first 48 hours. Hutchinson and colleagues needed extra home visits when their own equipment failed, which is the reason to budget support time rather than assume it. Log app opens, reminders shown, and taken or skipped events with timestamps, but not message content, and state that a taken tap is a proxy for self-reported adherence rather than a verified dose. Report the engagement curve weekly, define effective engagement in advance as adherence of the level the app targets, and report attrition with three denominators: enrolled, still active at week two, and completing week four. The expected failure is silent attrition in the first week from participants who cannot set the app up, which a support visit budget would catch.

The strongest critique is that the paper equates a proxy with the outcome. A fall in app use is compatible with several stories that the design may not own: residents who submit fewer but better reports, a seasonal lull, a fix that removed a bug which had caused repeated opens, or a change in the service the app connects to. Eysenbach's argument points to what is missing: the attrition curve itself, its shape, and the denominator. The paper needs to say how many people enrolled, how many were still active at each week, whether the fall was a cliff or a gradual decline, and who remained. It also needs a measure of the thing the app existed to produce, which Yardley and colleagues would call effective engagement, such as resolved reports, rather than time in the app. Brown and colleagues' demand characteristics give a further reason for caution: early use is inflated by being studied, so a decline from week one may be the fading of the study effect rather than a failure of the design. A supportable version would pre-state the threshold that counts as effective engagement, plot the curve with its denominators, report the outcome measure and where it came from, compare the app with a plausible alternative or a seasonally matched baseline, and describe who stayed and who left. The authors would then be entitled to the narrower claim that use declined and here is what followed from it.

The protocol claims an efficacy result from a design that cannot produce one. On this unit's argument the deployment will yield a claim about an app, two clinics and a six-week method together, so its reach is fixed by that design rather than repaired in the discussion. The team should say which conditions of use the design preserves and which it discards. Six weeks keeps the ordinary routine of the patients but may not survive past the first fortnight, and Brown, Reeves and Sherwood (2011) show that participants adjust their behaviour to what they take to be the investigators' expectations, so early adherence is a claim about the study as much as about the app. The team should also state how patients record use, because Carter and Mankoff (2005) found that the medium changed what participants recorded and how memory was jogged, so a paper diary and an in-app log are not interchangeable instruments. On the contested question, the team can claim with Rogers (2011) that the wild is where theory about reminder behaviour should be built, and it should accept with Kjeldskov and colleagues (2004) that a field evaluation of one system in two clinics does not automatically outperform a laboratory study that restores the conditions of use. Dourish (2006) would ask the team to stop treating design implications as the only currency of the study. What would settle the disagreement is a programme of comparisons across clinics and systems with the outcome measure fixed before data collection, and short of that the team should report what it fixed: the app version, who was recruited, which routine the deployment entered, what was logged, and where the study departed from plan.

The brief claims a needs analysis from a method that was never designed to deliver one. Gaver, Dunne and Pacenti (1999) present cultural probes as packs sent to return fragments of other people's lives, valued for inspiration and uncertainty rather than for extracted data, and Hutchinson and colleagues (2003) define a technology probe as carrying three goals at once, understanding needs, field-testing a technology and inspiring, while explicitly ruling it out as a prototype. On this unit's argument the study will produce a claim about a set of probes, two offices and a three-week method together, so its reach is fixed by that design. The team should say what it can claim, namely situated inspiration and critique, and what it cannot, namely the user needs that a scheduling tool must meet. That gap is the category error: judging the probe by an evidential standard it was never given, the error Boehner and colleagues (2007) trace to an unstated disagreement about what counts as valid interpretation. The team can partly repair it in two ways. It can relocate the probe's value to the design work around it, as Wallace and colleagues (2013) do, and it can treat the uncertainties the probes surface as generative resources, as Epp and colleagues (2024) and Giaccardi and colleagues (2024) argue, rather than as gaps to be closed. On the contested question, the originators would hold that the team should not make the probes evidential at all, since controlling them defeats their purpose (Gaver et al., 2004), while Bardzell and colleagues (2015) would demand a stateable knowledge claim and a critical reception of whatever the probes yield. Zimmerman and colleagues (2007) and Gaver (2012) would keep both positions arguable by treating the contribution as provisional and as an annotation of the probe material rather than a general law. What would settle the disagreement is a probe study that states in advance what it will assert and what would count against it and then has that assertion tested beyond the original project, or an agreed account of what a probe study owes its reader; Benford and colleagues (2005) offer a candidate vocabulary in expected, sensed and desired movements. Short of that, the team should report what it fixed: the probe content, who was recruited and who dropped out, how the offices differed, how the material was interpreted and by whom, and where the study departed from plan. Written that way, the study can contribute method and inspiration without pretending to have established the tool's requirements.

The figures describe use, not benefit. Sessions are a behavioural measure, and Perski and colleagues (2017) show that engagement is carried in two registers, experiential and behavioural, which answer different questions; Short and colleagues (2018) name sole reliance on system usage data as the limitation of current practice. So the report can say how often the app was opened, and it cannot say that patients were engaged in the sense that matters, still less that their condition improved. The falling average and the 71 per cent who opened it once are also a classic pattern, and O'Brien and Toms (2008) treat engagement as a process with a point of engagement, sustained engagement, disengagement and reengagement, which means the first month and the tenth are different states rather than one average. Yardley and colleagues (2016) supply the standard the trust needs: effective engagement is defined empirically as sufficient engagement with the intervention to achieve intended outcomes, and the relationship between use and the desired behaviour change must be understood rather than assumed. On this unit's argument the trust should name the clinical outcome first, then say what amount of use is supposed to be sufficient and why. That last step is where the literature divides. Sieverink and colleagues (2017) found that of the 28 studies that set a threshold for intended use, only six justified it, and Michie and colleagues (2013) show that a justification requires naming the active ingredient. Hoffman and colleagues (2023) show the use-outcome relation may not be monotonic, since the cluster with the lowest behavioural engagement had the greater symptom declines, while Borgnolo and colleagues (2025) show the setting can outweigh individual variables, with the service centre the strongest predictor of engagement. Reminder prompts are therefore an intervention with an assumed mechanism and no stated dose, and the budget decision presumes the relation they are meant to move. What would settle it for the trust is a pre-specified analysis: fix the outcome and the mechanism in advance, justify the intended dose, measure both registers with a validated instrument (O'Brien et al., 2018), and model the dose-outcome relation with the service or site as a level in the model (Borgnolo et al., 2025). In the coming year the trust should report who left and when, the distribution of use rather than its mean, the clinical outcome measured separately from use, and the justification for the dose it is trying to produce. If it cannot justify the dose, it should test it rather than assume a rise in sessions is progress.

The figures describe use, not benefit, and they describe a falling curve. Ninety-six of 300 residents were still using the system at nine months, and the report converts that into an engagement figure without stating what use was for, which is the move Eysenbach (2005) rejects: he argues that attrition measures should be reported alongside efficacy and that the characteristics of the subpopulation for which a system works should be described, not smoothed into a single percentage. The 47 per cent also conceals two different events, the 16 residents who never had a profile created and the 188 who stopped at some point, and Baumel and colleagues (2019) show why that distinction matters, since the same programmes deliver reported usage rates a median of 4.06 times the real-world rate when users are proactively recruited and assessed. On whether this is a property of the residents or of the deployment, the unit's evidence points to the deployment. Pratap and colleagues (2020) found median retention across eight remote studies ranged from 2 to 26 days, with clinician referral worth 40 additional days, compensation 22, having the clinical condition 7 and older age 4, and Geraghty and colleagues (2013) removed most attrition simply by adding live contact, cutting it from 48.1 to 10.8 per cent in an otherwise identical cohort. Beatty and Binnion (2016) found that most participant-level variables predicted nothing, and Linardon and Fuller-Tyszkiewicz (2020) state that no baseline characteristic reliably predicted attrition. So buying tablets to reach residents who 'chose not to participate' treats a design and support problem as a resident characteristic, and Kelders and colleagues (2012) add that being studied matters too, since randomised rather than observational design predicted better adherence. The contested question the purchase turns on is therefore whether attrition belongs to the person or to the arrangement, and what would settle it is the test Baumel and colleagues (2019) ran, comparing reported usage with an objective measure for the same system, scaled up: one system, one definition of use, one follow-up window, measured in homes that receive live support and homes that do not. Over the following year the partnership should report who left and when, the retention curve rather than a single figure, the reason each person stopped, taken from the residents themselves, and any clinical or care outcome measured separately from use, using tooling such as dropR to plot and test the curve (Reips et al., 2025). If it cannot say what amount of use was supposed to produce benefit, it should test that dose rather than assume that more tablets will produce it.

The trial standard asks the evaluation to demonstrate a measured change in a health outcome under conditions in which the intervention and its delivery package are stable and implemented with fidelity, which is exactly the situation Murray and colleagues (2016) describe as the condition under which a randomised trial is best undertaken. The service meets that condition, so the standard is not unavailable to it. What the standard cannot do is tell the service how patients live with the app, and it also rests on an assumption that Glasgow, Lichtenstein and Marcus (2003) show to be unsafe: that effectiveness follows naturally from efficacy across settings, populations and staff. Their remedy is to document reach, adoption, implementation and maintenance alongside the outcome, following the five dimensions of RE-AIM (Glasgow, Vogt and Boles, 1999), which together describe population impact rather than efficacy alone. A deepened account of use supplies the other half. Klasnja, Consolvo and Pratt (2011) argue that demonstrating behaviour change is often infeasible and unnecessary and that an evaluation should be matched to the behaviour-change strategies inside the system, such as self-monitoring, with studies that deepen understanding of experience. Blandford and colleagues (2018) explain why the two accounts are hard to combine, since HCI and health research differ over the nature of accepted evidence and the culture of reporting, not merely over method. Because the app is stable, the service can run a randomised comparison of the app against usual care, and because the app is a technology that may change, it should plan a continuous or adaptive element, following Mohr and colleagues (2013) and, if the intervention is a sequence, the SMART logic of Collins and colleagues (2014). The comparison can claim a measured difference in the primary outcome under the declared protocol; the use account can claim an explanation of how and why that difference arose or failed to arise, and it can explain a null result that the outcome measure alone cannot. Neither claim substitutes for the other, and the account of use should not be reported as evidence of benefit. Finally, whatever standard is chosen, Hall and colleagues (2024) show that obligations persist: harm from recruitment, monitoring of safety, exclusion of demographic or clinical groups, structural barriers to implementation, and regulatory ambiguity about software that is a medical device. The recommendation is a randomised comparison with a nested qualitative and usage study, analysed and reported separately, with the eight challenges of Hall and colleagues addressed in the protocol.

The plan should begin with the two documents the trial standard treats as a pair. For the write-up I would require the 25-item CONSORT 2010 checklist as the structural spine (Schulz, Altman and Moher, 2010), and for the record of what was planned before the deployment I would note that SPIRIT 2013 supplies a 33-item protocol checklist that asks authors to state in advance what they intend, including roles and data access (Chan et al., 2013), which addresses the SPIRIT finding that sponsor and investigator roles were inadequately described in 89 to 100 per cent of protocols. For the intervention itself I would require the 12 TIDieR items (Hoffmann et al., 2014): the brief name, rationale, materials, procedure, provider, mode, location, timing and dose, tailoring, modification, and fidelity against plan. For a home deployment I would add the domain items that the trial checklist omits. Signal and colleagues (2024) show how such additions are derived, since TIDieR-Rehab needed 12 new items to fit rehabilitation, covering the population, dosage and difficulty, personalisation and harms. The deployment equivalent should require the setting in detail, the recruitment route, the version of the device and its software, and every deviation from plan. This is also where Vanderdonckt and Vatavu (2025) bite, because they argue that the context of use is rarely stated clearly enough for reproduction and separate replication across cohorts, platforms, environments and contexts. McGrath and colleagues (2023) give the auditing method, since they scored 12 trials against the 12-item checklist and found a median of eight of 12 with none complete. The report cannot guarantee rebuildability. It can guarantee that the description is complete enough for another team to attempt the deployment, but it cannot make the claim true. Ioannidis (2005) explains why: small samples, small effect sizes and flexibility in analysis leave many published findings false whatever the writing, and a deployment with 24 households and one setting is exposed to all three. Nor can the report guarantee that the result will return, given that the Open Science Collaboration (2015) replicated 100 studies and found 36 per cent significant against 97 per cent originally, that Camerer and colleagues (2018) found 13 of 21 reproducing with effects about half the original, and that Prinz, Schlange and Asadullah (2011) found agreement in only 20 to 25 per cent of 67 drug target projects. The evidence that would show whether the plan worked is a trial of the checklist itself, joining the two measurements the unit already has: score the reports against the checklist as McGrath and colleagues did, then hand the reports and materials to independent teams and ask whether they can rebuild the deployment and reproduce the numbers, as Iarygina and colleagues (2026) did when they found they could fully reproduce 49 per cent of the CHI papers that shared materials. Report whether checklist scores predicted rebuildability. The objection the plan cannot answer is that replication may not transfer because the setting is a variable rather than a nuisance: Klein and colleagues (2014) found that variation in effect size across 36 samples was mostly attributable to the effect under study, and Hornbaek and colleagues (2014) found that HCI replications rarely analysed effect sizes at all, so a rebuildable report may still yield a result that differs for reasons the report cannot capture.

The proposal buys contact-free observation of daily life, which is exactly what the PlaceLab was built to provide: sensors integrated into the fabric of the setting, no researcher on site, and multi-day or multi-week data that a laboratory session cannot produce (Intille et al., 2006). The design has a direct ancestor, since the Aware Home was built with two identical independent living spaces so that inhabitants could live on one floor while technology was trialled on the other (Kidd et al., 1999). Instrumenting 30 occupied homes is a stronger version of the same bargain and inherits its costs. The residents become participants: they are recruited as volunteers, told what is recorded, able to withdraw data, and in the PlaceLab even their visitors had to sign consent forms, while one participant asked for audio from personal calls to be removed before release (Intille et al., 2006). Consent in an occupied home runs from the front door to the furniture, and the six-month term means the study cannot end by residents leaving. The proposal also commits the council to the place position on what a living lab is. Treating 30 occupied homes as instrumented places means the residents are the setting's observed occupants, not the co-creators that the network and methodology readings describe (Leminen et al., 2012; Dell'Era and Landoni, 2014). On measurement, the council should adopt an outcome-first design rather than a usage report: Veeckman and colleagues (2013) show how characteristics can be linked to outcomes, and Ballon and colleagues (2018) show both that an impact evaluation can be built and that the pragmatic nature of living labs makes attribution hard. Schuurman and colleagues (2016) warn that the return was greatest when a project moved from concept towards prototype, so a six-month field deployment may sit past the stage where living lab methodology pays most. The limit that cannot be removed is that a resident cannot be both a participant and a naive occupant. Intille and colleagues (2006) accept that moving people out of their own homes reduces the complexity of the setting and may change the behaviour observed, and that objection applies to instrumenting homes in place. The council should therefore run the same system in a lab apartment with temporary occupants, measure both, and report the difference as the size of the setting effect rather than pretending it is absent.

The proposal treats the home as an instrumented place and the resident as the user. The unit's evidence says that description is incomplete. Grinter and colleagues (2009) find that a home network runs on two intertwined kinds of work, the technical work of keeping the system going and the collaborative social work of the household, so a study that logs one resident sees neither. Neustaedter and colleagues (2009) show that in 41 of 44 families the mother was the primary scheduler, so the person the system notifies may not be the person who manages the arrangement. Recruitment should therefore be at household level, with the arrangement described, and the outcome should be counted at the level the routine is organised around, with the household's own account reported beside the individual result. That is what would settle whether the household or the individual is the right unit, and the literature does not settle it. The moral economy has to be settled first. Woodruff, Augustin and Foucault (2007) found acceptability decided by religious observance rather than usability, and a motion sensor was rejected because it made the person affect the world directly. Koh and colleagues (2025) recorded privacy acceptability before and after four weeks of sensing and found the judgement forming during the study, so consent at the start is a starting point rather than a settlement. The memo should name what may never be inferred from the sensors, and who may see it. The study must also budget for support, because Grinter and colleagues (2009) show the upkeep falls on residents and fails often, and for an end, because a household reorganised around a reminder system cannot simply be returned to its earlier state. Finally, the missing comparison is one that holds the technology constant and varies only how far the design defers to the existing routine. Taylor and Swan (2005) recommend supplying resources for households to build their own arrangements, while Consolvo and colleagues (2008) deployed a system designed to change routine; nothing yet tests the two against each other, and that comparison is what would.

The proposal treats the resident who signs as the person the study measures. Friedman, Kahn, Borning and Huldtgren (2013) separate direct from indirect stakeholders, and this deployment manufactures the second kind: partners, children, visitors, carers, and every resident whose device the access point logs. Those people did not enrol and cannot withdraw, so the ethics have to be settled before installation, when the choice of what the camera sees, what the sensors record, how long records persist and who holds the controls can still be changed. That is why the consent form cannot discharge the work. Shilton (2009) asks for control over what you share, with whom and for how long, and a signature supplies only the first; Luger, Moran and Rodden (2013) show that the document itself is beyond what a functionally literate adult can understand, and Caine and Hanania (2013) found no patient who wanted to share everything with every recipient, so preferences are granular by data element and recipient. Consent should therefore be indexed by recipient and modality, and treated as revisable, since Wang, Tan and Munson (2026) found older adults' preferences bypassed by whoever acquires and manages the device, often with protective intent. Anonymisation promises less than it appears to. Sweeney (2002) defines k-anonymity as a formal condition, and Narayanan and Shmatikov (2008) show that a released dataset can be attacked with a few items of background knowledge even when the data have been perturbed. The missing comparison is a deployment that varies only the consent procedure, granular against the standard form, and measures both what participants can state about what is recorded and who receives it, and whether they narrow their sharing during the study. That would settle whether the failure is implementation or the instrument itself, and no study in this unit has run it.

The memo should propose a micro-randomised trial rather than a two-arm comparison. Klasnja and colleagues (2015) supply the design: at each decision point at which the system may act, it randomises whether to act, so the effect of support on a proximal outcome can be estimated and its moderation over time and context modelled. That is what an adaptive programme needs, because the question is not whether support works but when it works, and Bidargaddi and colleagues (2018) show the design running at scale inside a live product, with a small effect, 3.9 per cent on next-day self-monitoring, that attenuated over the study. Before any randomisation the service must write down the specification: the states that matter, the decision points, the tailoring variables and the decision rules, which are the components Nahum-Shani and colleagues (2018) set out, since randomising a rule chosen for convenience estimates the effects of that choice and nothing more. The sensing must be presented as provisional. Trifan and colleagues (2019) find that passive smartphone outcomes lack correlation with clinical knowledge, so the memo should name the outcome the service actually cares about, such as weight or HbA1c, and treat the sensor signals as timing variables rather than as the measure of benefit. The horizon must outlast the programme. Hekler and colleagues (2013) found that maintainers and nonmaintainers diverged in shape, with nonmaintainers beginning to drop at week 10, and Kwasnicka and colleagues (2016) report that maintenance is explained differently from initiation, so a study ending at 12 weeks measures adoption. For pooling, the memo should report follow-up length, level of human support and the definition of engagement as moderators, because Linardon and colleagues (2019) found larger effects where a cognitive behavioural app came with guidance and reminders, which means the effect is partly a property of the service. The strongest objection is that a positive proximal effect is not the effect of the intervention: the estimate is conditional on the rule and decays with exposure, and Park and colleagues (2026) describe even their learned policy as hypothesis-generating and requiring prospective validation, so the adaptive trial relocates the generalisation problem rather than curing it.

A strong answer names mechanisms rather than features. The first likely failure is the disparity between who does the work and who benefits. The rota requires every clinician to keep availability current, while the benefit accrues mainly to the person allocating shifts, so the record decays fastest among those who gain least. The design response is to make the record a by-product of work the clinicians already do, for instance by generating availability from swaps they negotiate anyway, and to give every person a visible direct benefit, such as seeing who is on call at the other site. The second likely failure is exception handling. A system that enforces the standard rota cannot represent the informal swaps that keep a service running, so staff work around it and the record diverges from practice. The response is to design the exception path first, with a cheap, explicit way to record a swap and its reason. Awareness is the constraint that follows from the second section. Show who is on call, at which site, and whether they are reachable, and avoid a persistent, individually attributable record of response times, which turns coordination information into performance monitoring. Evidence would be the proportion of shifts needing manual correction, the volume of off-system negotiation left in messages, and interviews with the staff who enter least, because non-use is a finding rather than a failure.

The first decision is who the subjects are. The guidelines hold that the primary question is who counts as a subject, and that the greater the vulnerability the greater the duty to protect, which places posts about mental health among the groups they single out, alongside downstream harm and content that suggests risk to the author. Scraping does not settle consent. A public post was written for an imagined audience of fellow members, and context collapse means the researcher's audience is wider than the one the author pictured, so the team must argue why the analysis could not have been run with members' knowledge. Defensible choices exist. The team should decide in advance what will never be quoted, whether searchable strings will be altered or withheld, how long identifiable material will be kept and who can reach it, whether people will be linked across posts, and what it will do if a post indicates immediate risk, which the guidelines treat as a question answered differently under different professional and national obligations. Where a community has moderators or a governance process, asking them is cheap and informative. The study is defensible if the team can say what harm could follow from publication and what it did about each risk, and if its claims match the data: language patterns among self-selected posters cannot stand in for prevalence or diagnosis.

The brief treats the procedure as the thing to capture, which is the premature settling Bannon and Schmidt (1989) warned against; the object should instead be the cooperative work of handover and the articulation work that holds it together (Schmidt and Bannon, 1992). That reframing changes the test: not whether staff follow the procedure, but whether the work holds together across shifts, for whom and at what cost. The nursing culture around a handover also carries the mismatch Grudin (1988) places at the centre of failure, since the person who enters the data is often not the person who benefits. Schmidt and Simone (1996) give the trust a design vocabulary, because coordination mechanisms such as shared classifications, protocols and routines are the artefacts in which articulation work is deposited, and the object should be to support those rather than to dictate a script. The system is also sociotechnical and political: Kling and Scacchi (1982) argue its effects depend on the organisational context, and Kling (1991) warns that the same mechanism that coordinates also controls, so who sees the handover record and how it is audited is a design decision about accountability, not a neutral default. Procedure cannot carry the work because Suchman (1987) showed that plans are resources people use and repair in situ, and Hughes and colleagues (1992) showed that a rich account of work does not convert smoothly into design. The trust should therefore treat the object as generative, following Schmidt and Simone (1996), while acknowledging Ackerman's (2000) social-technical gap, that the social requirements exceed what any formal system supports, and Kuutti's (1996) claim that the unit of analysis is mediated collective activity rather than an individual's task. To let a reader judge the contribution, the trust should state in advance what the system will support, what it will not, and how it will be evaluated on whether the work holds together rather than on compliance alone.

The managers have made an adoption prediction and have mis-specified its cause. On this unit's argument, whether the handover system is used depends on the distribution of incentives, on critical mass and on the gap between prescribed and actual work, so a mandate is an organisational lever and an interface is not. Grudin (1994) argued that groupware failures were not technical but failures to meet eight challenges created by the difference between software for a person and software for a group; the first challenge is the disparity between who does the work and who benefits. A handover record may chiefly serve the manager and the incoming shift rather than the nurse writing it, which is the incentive problem in its ward form. Markus and Connolly (1990) sharpen it: a tool whose benefit is collective and conditional on colleagues has an adoption problem that no single-user design heuristic solves, so the design team cannot usability its way out. Markus (1987) explains why a mandate was the managers' instinct and why it is not free. Interactive media have universal access as a public good and reciprocal interdependence, so a discretionary handover log is vulnerable to start-up failure; Grudin and Palen (1995), asking whether a mandate was the route past that threshold, found near-universal adoption achieved without one, through a versatile product, organisation-wide infrastructure and peer pressure among colleagues, so a handover mandate may still buy compliance where that pressure is absent but the workaround remains the usual price of a tool nobody wants. The trust should therefore do two things. Before go-live, it should give every role a direct benefit (Grudin, 1994) and design the process of handover, not only the screen (Bikson and Eveland, 1996). After go-live, it should expect repair rather than completion, since Wagner and Newell (2006) found that a working system required continuous social work to reconcile the technology with local practice. The strongest counter-argument is that the artefact is a party to the outcome, not a bystander: Barley (1986) found the same technology produced different organisational structures, and Leonardi (2011) argues human and material agencies are imbricated, so a well-made handover form can change what staff are able to do. The narrower, defensible claim is that social organisation decides whether any benefit is realised while the design decides which reorganisations are available, and that is the version the trust should design and measure against.

The feed is doing coordination work, and that is exactly why it should not be handed to the director unexamined. Dourish and Bellotti (1992) show that the information holding shared work together is generated passively, as a by-product of activity in the shared object, and that users complained when a relevant trace was missing; on that account the fall in duplicated referrals is consistent with the feed working as a coordination mechanism. It cannot support a claim about individual productivity. Awareness of the character of other people's actions lets a person structure their own activity and avoid duplication, which is a property of the arrangement rather than a measure of one clinician's output, and nothing in the feed shows whether an edit or a view improved a discharge. On separation, the unit's answer is that the coordination benefit and the monitoring risk travel on the same signal, which is why Hudson and Smith (1996) call the privacy and awareness tradeoff fundamental; the practical question is where the board sits on it. Symmetry, decay and aggregation are the levers, and the director's request is a request to make the feed persistent and unaggregated for a new audience. The contested question is whether design can separate the two: Erickson and Kellogg (2000) answer with translucence rather than transparency, Ackerman and Starr (1995) with indicators that summarise activity instead of logging it, and Prinz (1999) with an awareness infrastructure in which access control and reciprocity are configurable, while Schmidt (2002) argues that monitoring and displaying are mutually dependent, so a feed read by managers changes what clinicians display. What would settle it is measurement rather than argument: hold the discharge work constant and vary symmetry, decay and aggregation, then record coordination outcomes such as duplicated referrals and unclaimed handovers alongside monitoring outcomes such as whether the feed is consulted by people with authority over those they display. Prinz's (1999) design is the nearest existing instrument, and Cadiz and colleagues (2002), whose interface reached 13,000 installations in a year, are the nearest measure of the attention cost that such feeds impose. Over the year the trust should make the feed symmetric, let traces decay, aggregate before display, publish who can see the data, and report the reasons clinicians stopped using it.

The firm is proposing to remove proximity from its most coupled work and leave it in its least coupled work, which is close to the wrong way round. Distance does not cost through bandwidth. Olson and Olson (2000) reduced successful remote work to four conditions, high common ground, loosely coupled work, collaboration readiness and technology readiness, and only the first two bear on this decision. Kiesler and Cummings (2002) locate the distance effect in informal, spontaneous encounter and in weakened identification with the larger group, which no scheduled call supplies. Cramton (2001) gives the mechanism for the platform team specifically: dispersed members assume a shared context that is in fact local, and the resulting gaps are systematic and misattributed. The quantitative cost is real; Herbsleb and Mockus (2003) found distributed work items took about two and a half times as long as colocated ones and involved more people, and Herbsleb and colleagues (2000) found delay rose with perceived unwillingness to help under load. By contrast the evidence of benefit is narrow: Bloom, Liang, Roberts and Ying (2015) found a 13 per cent performance gain and halved attrition for individual call-centre work, and Choudhury and colleagues (2021) a 4.4 per cent output gain for examiners. The support function resembles those cases; the platform team does not. The contested question is whether measured productivity answers the distance claim. Olson and Olson (2000), Cramton (2001) and the software-engineering studies say the coordination penalty is durable; the productivity studies say it is conditional; Emanuel and Harrington (2023) and Gibbs and colleagues (2023) argue the productivity figures are confounded, since selection decides who works remotely and because hours rose while output fell in one large study. What would settle it is a design holding the work constant while varying proximity and measuring coordination outcomes, cross-site delay, the number of people a work item routes through and the bridges in the collaboration network, alongside output and selection. Over the coming year the firm should keep the platform team partly colocated, invest in the mitigators Espinosa and colleagues (2007) identify, shared knowledge of the team and presence awareness, and measure delay and handovers for both functions rather than headcount or hours.

The project is treating a pipeline problem as a quality problem. Halfaker and colleagues (2013) describe this configuration exactly: as Wikipedia grew, its community used its primary quality control mechanism more restrictively, the share of desirable newcomers whose first-session edit was reverted rose from 6.1 to 18.2 per cent between 2006 and 2007, two-month survival of those newcomers had stood at 25.6 per cent in early 2006, the survival rate fell to 11.7 per cent and did not recover, and active editors declined from a peak of 56,400 in March 2007. Stricter first-edit rejection is therefore likely to deepen the flattening the maintainers are trying to reverse. Butler (2001) adds that growth raises cost when the resources for communication do not scale, so more members can make the project less sustainable rather than more. Choi and colleagues (2010) give the alternative: welcome messages, offers of assistance and constructive criticism slowed or reversed the general decline in newcomers' edits, while invitations led to steeper declines, so the maintainers should replace the automated welcome with personal assistance and useful criticism. Lampe and Johnston (2005) show that feedback on a new member's first contributions, not instruction, changes what they do next. On cost, Cosley and colleagues (2007) raised edits by roughly four times by routing tasks to contributors instead of suggesting random pages, so the project should route the backlog to people who can do it and cut the time spent finding work worth doing. Motivation should not be treated as one lever: Nov (2007) found fun and ideology the strongest of eight motivations but no significant link between stated ideology and contribution, and Crowston and Fagnot (2018) found initial, sustained and meta motives differ, so onboarding and retention need different appeals. The contested question is whether the newcomer's trajectory is set on arrival (Panciera et al., 2009) or written by the receiving community (Choi et al., 2010; Lampe and Johnston, 2005), with Antin and Cheshire (2010) placing the pipeline upstream of the first edit. The project cannot settle that with its own logs alone, but it can test its changes: define newcomer cohorts, assign the receiving practices at random where the workflow allows, and follow each cohort with trace measures, time to first accepted edit, reversion of first-session edits, two-month survival and return after criticism, so that selection and treatment are separated.

A strong answer begins with the affordances boyd (2010) named: persistence, replicability, scalability and searchability. Removing old posts from strangers' feeds lowers scalability and searchability for those posts, and finer audience controls partly restore the collapsed context, but persistence is untouched, because the content remains archived, and replicability is untouched, because anything shared can still be copied on. The redesign should not be framed as repairing user skill: Marwick and boyd (2011) and Duguay (2016) show users already deploy targeting, concealment and audience separation, so the burden was structural rather than a competence gap. The audience-size display answers the measurement gap Bernstein and colleagues (2013) quantified, in which users guessed their audience was about 27 per cent of its true size, and should correct that estimate, but showing a number does not by itself restore a context, and Vitak (2012) found that broader, more diverse audiences narrow disclosure and drive use of privacy controls, which the redesign can support. The harms it will not fix include the unequal cost Duguay (2016) documented, since a member whose identity is not accepted by the default audience still needs to separate audiences deliberately and continuously, and the anchoring Zhao and colleagues (2008) described, since the profile stays tied to a real name and to group and consumer signals. The team should test with the manipulation the unit proposes: randomise whether users see their true recent audience, hold the platform constant, and measure strategies and disclosure before and after, using Bernstein's estimate-gap measures rather than intention. A within-person comparison across platforms would separate the structural claim from the skill claim.

A strong answer begins from the unit's claim that crowd and platform work are one coordination problem, the allocation of information and control, so the right question is what each change does to disclosure rather than to the legal label. Showing terms before acceptance addresses exactly the asymmetry Rosenblat and Stark (2016) identified as fundamental, since they argue the information and power asymmetries produced by the application, not the contract, are what structure control, and they show the rhetoric of entrepreneurship obscuring it. Portable reputation and a worker-run review of requesters answer Irani and Silberman (2013), whose Turkopticon supplied the disclosure and accountability the platform withheld and turned an anonymous crowd into a collective. Kellogg and colleagues (2020) predict the limit: the platform still records and rates, restricting and recommending through its algorithms, so a redesigned disclosure is a change within the six mechanisms rather than a removal of them, and Rani and Furrer (2021) show the same architecture assigning, monitoring and evaluating across many countries, which is why they demand transparency in architecture and algorithms. The redesign will not fix the low pay, isolation and exhaustion that Wood and colleagues (2019) found coexisting with flexibility, because those follow from pay-setting and job design rather than from disclosure, and it will not fix the inequality Ticona and Mateescu (2018) documented where platform visibility serves companies and clients. Nor does it touch the worker response Cameron (2022) described, since engagement through games is not the same as a say in the rules. The team should test with the manipulation the unit specifies: randomise disclosure to workers, hold the interface otherwise constant, and measure task acceptance, realised pay and dispute outcomes, using the worker-side networks Gray and colleagues (2016) document as a check on whether information already spreads outside the app.

A strong answer starts from the unit's claim that moderation is governance, so the team should ask what each element does to authority rather than only what it does to behaviour. Publishing the rule breached and the deciding party meets the standard Suzor and colleagues (2019) set, since they argue that meaningful transparency must be a component of a communicative process of accountability to independent stakeholders, and they name exactly this information, together with access to disaggregated data, as what a person needs. The appeal panel goes further, because Crawford and Gillespie (2016) show that the flag fixes the vocabulary of complaint before any complaint is heard, so a route to challenge a decision changes who can answer back. The classifier addresses scale, which is the justification Gillespie (2020) identifies for automation, and the team should note his objection: the question of what should be removed is not the question of what can be detected. Chandrasekharan and colleagues (2019) supply the cautionary result, an accurate classifier whose flagged comments stayed online in 98.3 per cent of cases, so detection is not the binding constraint. Keeping the rules platform-authored leaves the deepest issue untouched, because Fiesler and colleagues (2018) show that communities write and enforce their own regulations across 100,000 subreddits, and Matias (2019) shows that making rules visible raised newcomer compliance by more than 8 percentage points and newcomer participation by 70 per cent, so community authorship is where the norms remedy actually lives. The redesign will not settle whether the platform or the community holds authority, which is the position Gillespie (2018) and Roberts (2019) press when they locate the deficit in decisions taken out of view and in the hidden labour of moderators. Nor will it fix what Kiene and colleagues (2016) show communities can do for themselves, since adaptation by the governed is not authority granted to them. To test the redesign, the team should randomise whether a removal is explained and whether an appeal is offered, and measure both later compliance and the affected person's judgement that the decision was legitimate, combining the randomised template Matias (2019) provides with the accountability outcome Suzor and colleagues (2019) specify, and comparing platform enforcement with community enforcement on the same rules.

The team should begin by separating what the literature measures. The finding that false news spreads faster than true news rests on a diffusion advantage across roughly 126,000 stories, and Vosoughi, Roy and Aral (2018) also found that robots spread true and false news at the same rate, so the mechanism is human sharing. Diffusion is not reach. Allcott and Gentzkow (2017) found the average adult saw one or a few false stories, and Guess, Nyhan and Reifler (2020b) found untrustworthy websites were a small share of information diets, concentrated in a subset with strong pro-attitudinal preferences. The blocking measure is therefore justified by nothing in the unit: no cited study tests a platform block, and the closest analogue, four weeks of randomised deactivation, reduced factual news knowledge as well as polarisation (Allcott et al., 2020). The accuracy prompt has direct support. Pennycook and colleagues (2020) nearly tripled truth discernment in later sharing intentions with a single pre-sharing accuracy question, and the mechanism is a failure of reasoning rather than partisan bias, since reflection predicted better discernment even for aligned headlines (Pennycook and Rand, 2019). The literacy lecture is the measure whose evidence is most conditional: Guess and colleagues (2020a) raised discernment by 26.5 per cent in a representative US sample and 17.5 per cent in an educated Indian sample, but the gain did not persist in India and there was no effect at all in a rural northern Indian sample. The team should ask whether its students resemble the samples that improved. To test the result, the team should pre-register the outcome, randomise the prompt across portal users or by period, measure discernment in sharing decisions rather than belief alone, and track behaviour over months, since effects that vanish after weeks are a known pattern in this literature (Guess et al., 2020b). It should also avoid claiming a mental-health benefit, because the two strongest designs disagree: deactivation improved subjective wellbeing in four weeks (Allcott et al., 2020), while the staggered arrival of Facebook worsened student mental health over years (Braghieri, Levy and Makarin, 2022), and the difference is the counterfactual. Finally, the team should not treat user education as the whole answer. Rathje, Van Bavel and Van der Linden (2021) found out-group language the strongest predictor of engagement, and Brady and colleagues (2023) argue that engagement-optimising algorithms amplify such content, so the incentive to produce it sits outside any individual student's judgement.

The department should begin with the claim that trace data are made rather than found: a dataset is the residue of product decisions, and what the platform recorded fixes the sampling frame before any analysis begins (boyd and Crawford, 2012). From that follow three requirements it can write into the policy. First, every project should state what population, if any, the data can speak about, because platform users are not a sample of anything in particular and scale does not repair the inference (Tufekci, 2014); the policy should ask for a written sampling statement, not a size threshold. Second, access determines what can be studied, so projects that depend on cooperation from a platform should say what was not observable without it, and where that is not possible they should consider an audit design, remembering that audits themselves concentrate on what researchers can reach without permission (Bandy, 2021; Lazer et al., 2020). Third, the consent question cannot be answered by the word public. Hudson and Bruckman (2004) found that occupants of a technically public chatroom objected when studied, being removed from 63.3 per cent of the rooms in the announced conditions against 29 per cent of the control and finding only 4 of 766 occupants willing to opt in; Fiesler and Proferes (2018) found the majority of surveyed users thought tweets should not be used without consent, with attitudes varying by who conducts the research and what it is about; and Zimmer (2010) argued that technical availability settles nothing about ethical status. The policy should therefore require a consent or justification decision for anything beyond the four conditions Bruckman (2002) set out in 2002: officially public archiving, no password, no site prohibition, and a topic that is not highly sensitive. On the contested question, the department should not expect the literature to hand it a rule. Vitak, Shilton and Ashktorab (2016) surveyed 263 researchers who wanted clearer norms and disagreed about what those norms should be, and franzke and colleagues (2020) hold that internet research evokes more than one ethically defensible response, so that the best instrument available is guidelines rather than recipes. A third position holds that the obligation attaches at collection: if the data were gathered without consent, later anonymisation does not repair it, and the regulatory framework does not reach these people (Metcalf and Crawford, 2016). The department can write a policy that requires a named owner, a stated sampling frame, a documented consent decision and a storage and sharing plan (Zook et al., 2017), while accepting that the hardest cases will need judgement. To learn whether it works, it should record decisions and outcomes rather than attitudes, since the existing evidence reports what researchers believe and what users expect, and not what is done with a dataset once it is held.

A strong answer picks one method and defends it, then treats each rejected alternative as a real option rather than a straw figure. The obvious choice is a longitudinal qualitative study: interviews with members of several households, plus observation of the device in use over a period of weeks, because the negotiation is a process that happens between conversations and would be invisible in a single sitting. The first alternative to reject is the controlled usability test. It would answer how quickly one person completes a specified task against a specified measure, which is a different question. What is lost by rejecting it is standardisation and comparability: the study cannot say that one design is faster than another. The second alternative is a survey of device owners. It would answer how common a particular arrangement is across a large sample. What is lost is access to the reasoning behind an arrangement, and a survey also risks asking participants to report a negotiation they may not have noticed. Greenberg and Buxton argue that the choice of methodology must arise from the research question, which is the test to apply here. The answer should also say what the chosen method cannot do: a small number of households cannot support a claim about how common any arrangement is, and the study must not imply that it can. Causal claims about which household arrangement works best are also outside what the method can establish.

The plan is wrong at each of its four steps, and the answer should say why rather than simply marking them. Preregistering a design study wholesale is the first problem, because a design study produces an artefact through iteration and cannot fix its comparisons in advance. The honest move is to preregister the parts that can be fixed, such as the evaluation of the finished artefact, and to label the design work as exploratory. Second, taking an effect size from a four-week pilot follows the practice Leon, Davis and Kraemer warn against: a pilot is not a hypothesis test, and an estimate from a small sample is too imprecise to plan a power calculation. The pilot should instead be reported as a feasibility result, giving quantified recruitment, retention and assessment figures. Third, deferring ethics approval until after the pilot is the most serious error, because the pilot is itself research with participants and cannot be run before approval. The application goes in first, covers the pilot and the main study, and is amended later if the design changes. Fourth, calling every analysis confirmatory is false. Analyses that were decided after seeing the data are postdictions, and the honest account labels each analysis as planned or emerged, which is the distinction Nosek and colleagues draw. A revised plan states which analyses are confirmatory, which are exploratory, and what the pilot changed.

The sentence makes a gap-spotting move: it asserts an absence and formulates the question from it. Sandberg and Alvesson (2010) found gap spotting to be the most common way researchers construct questions across paradigmatic camps, and their review of 52 articles records that dominance as surprising, since theory becomes influential when it challenges assumptions. The cost at the panel is that the sentence is silent about the field: a gap that matters and a gap that exists for a reason look identical from outside, so the student cannot answer the obvious question of why nobody has done this. Two handbooks explain why the move is taught, since Bell (1993) treats the review as the activity through which the question is refined and Booth (2003) organises a project around a question and a problem rather than a topic, but neither distinguishes an absence that was demonstrated from one merely not found. Reframed as a problematisation, the question would name an assumption. Alvesson and Sandberg (2011) propose problematisation as a methodology for identifying and challenging the assumptions underlying a literature and generating questions from a competing assumption. Here the candidate assumption is that alerting is the difficult part of a notification, so the reframed question asks what happens when the alert is not perceived, or how people negotiate a reminder that a device issues. Locke and Golden-Biddle (1997) show why this matters for the writing: authors construct an opportunity for contribution by first establishing coherence with prior work and then problematising it, so contribution is made in the review rather than found in the topic. Contribution also has a type. Whetten (1989) sets out the ingredients a reviewer looks for, the factors, the relationships among them and the rationale that explains why those relationships hold, and work that changes only the first two is a refinement. Corley and Gioia (2011) add that a contribution varies in originality, from incremental to revelatory, and in utility, from scientific to practical. Sutton and Staw (1995) sharpen the test by noting that references, data, variables, diagrams and hypotheses are not theory, so the student's explanation must survive the removal of the model diagram and the hypothesis list. Wacker (1998) supplies the checklist a problematised question must still pass: conceptual definitions, a bounded domain, relationship building and predictions. Finally, the student should not assume that the reframed question is stronger, since nothing in this literature matches the move to the outcome. The honest claim is that problematisation is an argument with good reasons and no direct evidence, and the study that would settle it would compare reviewer judgements of two versions of the same proposal that differ only in how the question is framed.

The student is making an artefact promise and dressing it as an empirical finding. Wobbrock and Kientz (2016) set out seven contribution types and describe how each is judged, and the two at issue here are distinct. An empirical contribution stands on the importance of the finding and the soundness of its method, which means a design that could have produced the opposite result; twelve participants in a single comparison give only a weak basis for a general speed claim, so the empirical promise is larger than the evidence the design can produce. An artefact contribution is judged on the artefact and on the evidence about its use rather than on a measured effect, and that is the standard the gaze technique can actually meet. If the student claims the artefact, the proposal should plan against a design process: Peffers and colleagues (2007) give six activities, from problem identification through design, demonstration and evaluation to communication, and each milestone should serve one of them. March and Smith (1995) make the criterion explicit, since design research is judged on usefulness and performance against requirements rather than on the truth of a proposition. Where the knowledge is design knowledge rather than a measured effect, Zimmerman and Forlizzi (2014) and Gaver (2012) argue that it should be judged by its ability to reframe a problem and to generate new design possibilities, and Hook and Lowgren (2012) add the intermediate level of a strong concept. The contested question is whether the type fixes the standard at all: if Oulasvirta and Hornbaek (2016) are right that problem-solving capacity is a common currency, the two versions would be compared on what each enables and the mismatch would matter less than this unit claims. What would settle it is not a better prototype but a matched design, in which the same proposal is presented twice, differing only in the type claimed, and judged blind by reviewers with the quality of the underlying work held constant.

The thesis is that a review earns its place by making the contested question visible and by saying which studies are strong rather than by listing who wrote what, and that the choice between systematic, scoping and narrative review is a choice about what the review can claim. The student's chapter, as proposed, fails on both counts: describing forty papers in turn is the author-centric review that Webster and Watson (2002) contrast with a concept-centric review organised through a concept matrix, and relabelling it makes the mismatch worse rather than better. Snyder (2019) holds that the question and purpose determine the strategy, not the reverse, so the label must follow the question. Rowe (2014) defines a review as work that synthesises past knowledge, identifies biases and gaps, and proposes future directions, and grades the requirement: a review aiming at explanation must be highly systematic, while a review aiming at understanding need not be, provided coverage is good. Since the student's question concerns how older adults negotiate reminders, a conceptual question rather than an effect estimate, a semi-systematic or narrative review is defensible. Munn and colleagues (2018) give the decision rule for scoping: it is appropriate to identify gaps, scope a body of literature, clarify a concept or investigate research conduct, and the student's question about a contested assumption fits the clarify-a-concept indication. Arksey and O'Malley (2005) set out the first scoping framework, including the purpose of testing whether a full systematic review is warranted, and Levac and colleagues (2010) require a clear link between purpose and question, a balance of feasibility against breadth, and a stated stopping rule. On auditability, Page and colleagues (2021) supply the twenty-seven item PRISMA 2020 checklist, Tricco and colleagues (2018) supply the scoping checklist with twenty essential and two optional items, and vom Brocke and colleagues (2009) require the search process to be documented so a reader can judge exhaustiveness. Brereton and colleagues (2006) show that a method built in medicine needs re-specification in a computing field, where indexing is weaker and abstracts cannot be trusted to judge relevance. On appraisal, Baumeister and Leary (1997) insist that a review which merely describes a series of studies warrants no publication and that a critique of the evidence is central. The contested question is whether the systematic review is a gold standard or whether the hierarchy is spurious, with Greenhalgh, Thorne and Malterud (2018) holding the complementary view, and Rowe (2014) and Baumeister and Leary (1997) grading the requirement by goal. My recommendation: keep a narrative or scoping label, state the purpose, organise the chapter around the contested question rather than around the papers, document the search so it can be appraised, and say for each landmark study what it found and how strong it is. The student should not relabel the chapter a systematic review to look rigorous, because the label is a promise the method cannot keep.

Start by making the purpose explicit, because Greenberg and Buxton (2008) show that the same evaluation fails when it is run by rule rather than by the problem: a laboratory comparison can answer how quickly and accurately a technique is used, but it cannot show that anyone would adopt it, and run to verify a prototype it can suggest the design is worthy instead of critiquing its use. The alternatives therefore have to be named. Charness, Gneezy and Kuhn (2011) give the trade-off for the design itself: the within-subject version buys power at the cost of the confounds that repeated measurement brings, while a between-subject version is more conservative and needs more participants for the same precision. The wider rejected alternatives belong to the repertoire Cairns and Cox (2008) set out, so the student should say why a field study or an interview study was not the method, and what standardisation that choice gives up. The measures need the same treatment. Hornbaek and Law (2007) found correlations among usability measures generally low, so error rate, completion time and satisfaction are three partly independent results rather than one, and the student should nominate one as primary and say what the others are for. The criteria should be fixed in advance, as Hartson, Andre and Williges (2003) propose: state the purpose, then report validity, thoroughness, cost and downstream utility rather than inheriting the word better from an earlier paper. The design forbids the general claim that gaze-based selection is better: with 16 participants and a laboratory task it can support a bounded statement about accuracy and speed against a named baseline, not adoption or preference in use. The strongest objection is Gray and Salzman's (1998), that small design decisions in comparison experiments have repeatedly carried more weight than the results they produced, and Wixon's (2003), that the question the student is asking is the laboratory question rather than the practitioner's question. The reply is that the purpose has now been stated, so the claim is bounded by it rather than hidden.

Begin with the inference, because Messick (1995) makes validity a property of the inferences and actions based on scores rather than of the questionnaire: the plan as written will licence a claim about change in a self-reported score after an hour of use, and not a claim that trust in the system increased, unless the student argues that the scale measures trust and that the observed change is attributable to the assistant. Cronbach and Meehl (1955) set the price of that argument: the student must say which theoretical network the construct sits in and what evidence beyond internal consistency locates the score in it. Bland and Altman (1997) then bound the questionnaire evidence: alpha describes whether the ten items cohere, with 0.7 to 0.8 satisfactory for comparing groups, and it is silent on whether the summed score means trust. The threats follow from the design. Campbell (1957) supplies the internal ones, since a single session with no comparison condition leaves history, maturation, testing and instrument decay uncontrolled, and the same paper supplies the trade, because tightening those controls by adding a control group and a second session would restrict how far the result generalises. Bracht and Glass (1968) name what that costs externally: population validity, since a university pool is an accessible population and not the target population of schedulers, and ecological validity, since an hour's use is a novel setting with sensitisation to the first questionnaire. Schmuckler (2001) warns that the second of those is loose unless the student says which dimension is at stake, the setting, the stimuli or the response measure. Podsakoff, MacKenzie, Lee and Podsakoff (2003) mark the measurement problem that no amount of analysis removes: both measurements come from the same instrument in the same setting, and Furnham (1986) adds that the evidence points to socially desirable responding as a relatively stable, multidimensional trait rather than an occasion effect, so a subgroup will move together. Cohen (1994) disposes of the decision rule, since p < 0.05 states the probability of the data given the null rather than the probability that the hypothesis is true, and answers neither what size of change matters nor how precisely it was estimated. The added evidence I would require is a validity argument for the scale in this setting, either published convergent evidence or a pilot that tests the items against a second measure of trust, and I would apply Messick's consequential conception because the plan proposes to act on the score as a design verdict. The strongest objection to my advice is Borsboom, Mellenbergh and van Heerden's (2004): a causal validation that manipulates trust and observes the score would settle more than any correlational accretion I am asking for, so my requirement risks collecting the wrong kind of evidence.

Begin with the purpose, because Leon, Davis and Kraemer (2011) define a pilot as an examination of feasibility and state that it is not a hypothesis testing study, that safety, efficacy and effectiveness are not evaluated in it, and that it does not provide a meaningful effect size estimate for planning later studies. The student's 1.4 second difference is therefore not a trend to be reported but an estimate the design cannot support, and reporting it invites the reader to treat an underpowered comparison as a finding. The powering plan repeats the error in a second form: Kraemer and colleagues (2006) find that guiding power calculations from a pilot most likely aborts studies worth performing and preserves underpowered ones, and Teresi, Yu, Stewart and Hays (2022) explain why, since with small samples effect size, completion and adherence estimates carry large confidence intervals and most pilots should not be used to estimate effect sizes or provide power calculations. What the pilot should measure is the feasibility set: recruitment, retention, fidelity, acceptability, adherence and engagement (Teresi, Yu, Stewart and Hays, 2022), which Thabane and colleagues (2010) put as safety, recruitment potential, coordination and the team's experience with the method. Recruitment claims must also be stated in units, in the way Walters and colleagues (2017) report them, participants per centre per month with a retention figure, in a programme where only 56 per cent of 151 trials reached their final target and median retention was 89 per cent. On size, Teare and colleagues (2014) recommend at least 70 measured subjects, 35 per group, for estimating the pooled standard deviation, and Whitehead, Julious, Cooper and Campbell (2016) recommend 75, 25, 15 or 10 per arm depending on the standardised effect size, so a pilot of five per group cannot carry an effect-size purpose at all, and the efficient remedy is a larger pilot rather than an inflated main study. Before approval I would require the proposal to state that the pilot's purpose is feasibility, to name the feasibility outcomes and their decision rules in advance in the manner of the CONSORT extension (Eldridge and colleagues, 2016), to give a planning basis for the main study that is not the pilot's effect size, and to say what result would stop the project. The strongest objection is that this advice may leave the main study planned on an assumption borrowed from the literature, which is exactly what a pilot is often run to avoid, and the honest answer is that Teare and colleagues (2014) and Whitehead and colleagues (2016) show how a pilot designed for that purpose can inform the plan, so the objection argues about the pilot's design rather than against the caution.

Begin with the order of work, because Emanuel, Wendler and Grady (2000) show that informed consent is neither necessary nor sufficient and that requirements such as value, scientific validity, fair subject selection and a favourable risk-benefit ratio are properties of the design. The student has fixed the design, four weeks with twelve participants, and deferred approval to the end, which reverses the sequence the requirements impose; approval is a precondition of running the study, so the plan should be built backwards from the submission date. Second, the requirements attach to the design: twelve volunteers drawn from one group through their facilitator is a selection decision that has to be argued, and the burden on a participant, four weeks of continuous screen-time capture asked of people using mental health services, has to be stated and justified as a risk-benefit judgement. The Menlo Report, as summarised by Bailey, Dittrich, Kenneally and Maughan (2012), applies to research with the potential to harm humans whether or not those humans are direct subjects, so the section must cover the other people whose messages, names and activity appear in four weeks of screen recordings, what will be redacted, who holds the files, how long they are kept and what happens to the dataset. Doerksen and colleagues (2024) add the second authority: work with a community may meet guidelines, a community review board or an advisory board of that community, so the facilitator's role and any group-level consent have to be negotiated before recruitment, not after. The Economic and Social Research Council (2025) makes the scope explicit, requiring ethics issues to be considered across the lifecycle including archiving, future use and the linking of data, which places the published dataset and any later reuse inside the application rather than outside it. On the contested question, the honest position is the one the evidence supports: Mittelstadt (2019) argues that consensus on principles should not be celebrated when the field lacks professional norms, methods to translate principles into practice and robust accountability, Bietti (2020) warns against both ethics washing and ethics bashing, and franzke and colleagues (2020) treat ambiguity and disagreement as inevitable, so the application should carry its own reasoning rather than rely on a principle being self-applying. At the same time Hedgecoe (2015) shows, from two cases, that a committee's decision can reflect its institution's reputation, so a project working with a mental health group should treat approval time as a range and record the risk. The strongest objection to this advice is that it is expensive for a student project, and the literature supports the objection: Lofstrom and colleagues (2024) find that members reason well and apply the elements of ethical decision making, which is evidence for treating a committee as a competent reader rather than a hazard, and no source in this unit audits committee decisions against participant outcomes, so the institutional-interest position rests on two case studies and cannot size the risk it describes.

Begin with the anonymisation claim, because it is the one the student states most confidently. Sweeney (2002) shows that k-anonymity, the formal model behind the idea, guarantees only that each record is indistinguishable from at least k-1 others on the attributes treated as quasi-identifiers, and only under the deployment policies the paper describes; it is a conditional guarantee, not a property that deleting names confers. Narayanan and Shmatikov (2008) re-identified records in a released dataset of 500,000 users with modest background knowledge drawn from a second source, and Rocher, Hendrickx and de Montjoye (2019) estimated that 99.98 per cent of Americans could be re-identified from 15 demographic attributes even in incomplete data. Ten hours of interview transcripts about shift handover in one named hospital are richer than those attributes: the hospital, the ward, the shift pattern and the role are quasi-identifiers, and the incidents nurses describe may identify patients or colleagues. The proposal should therefore name a release model, such as controlled access or redacted extracts, state the auxiliary data a reader could use, and give a risk estimate. That estimate is the step researchers most often skip: Rodriguez and colleagues (2024) found that 92 per cent of their respondents used documented de-identification and anonymisation processes, but only 29 per cent had any knowledge of re-identification risk estimation and none used a methodology. Second, the consent claim is weaker than it looks. Acquisti, Brandimarte and Loewenstein (2015) find privacy decisions uncertain, context-dependent and malleable, so a form signed at the start predicts little about acceptance of open publication; Nissenbaum (2004) relocates the wrong from the absence of consent to the inappropriate flow of information away from the context it came from; and Fiesler and Proferes (2018) find that users want consent in specifics, varying with who is researching, how the work is disseminated and what it is about. Consent should name the release and the reuse, not just the recording. Third, the design answers the suggestion that a data protection impact assessment is unnecessary. The Information Commissioner's Office (2023) guidance makes the assessment the instrument that documents the processing, its necessity and proportionality, and the risks, and the regulator can be consulted where high risk remains; Voigt and von dem Bussche (2017) place data subject rights and the supervisory authority in the same frame; and Stalla-Bourdillon (2025) shows the emerging European test resting on a functional definition of personal data and on the burden of proof, which the student cannot carry without a written assessment. Cavoukian (2009) and Langheinrich (2001) supply the cheapest fix: decide at design time what must be collected and how long it is kept, since end-to-end lifecycle protection and privacy by default reduce the material that ever needs releasing. My advice takes the position that anonymisation still does work, but only as a documented and conditional claim, which is the position Sweeney and Stalla-Bourdillon support. The strongest objection is the one Rodriguez and colleagues document: quantified risk estimation is underdeveloped in practice, so demanding an estimate may ask for a number no available method reliably supplies, and the student should acknowledge that gap rather than conceal it.

Begin with the milestones, because they are the part of the plan most easily fixed. Turner and Muller (2002) argue that uniqueness, novelty and transience generate uncertainty, integration and urgency, so a plan that carries only dates has answered the urgency alone. Applied to this plan, none of the three milestones passes the test of being inspectable by another person: the end of a reading week is not a deliverable, and being ready to write is a judgement no one outside the project can date. Rewrite each as an artefact with a date and a recipient, for example that the interview schedule is approved, the ethics application is submitted and receipted, and a coded subset of three transcripts exists. Next, the estimate of five weeks for twenty four interviews is an estimate formed from intention, exactly what Flyvbjerg (2006) documents as systematically optimistic, since forecasts drawn from the components of a plan suffer optimism bias and strategic misrepresentation. The remedy he proposes is reference class forecasting, which estimates from the outcomes of comparable completed projects, so the student should find completed studies recruiting care-home staff in a comparable setting and build the calendar from what they achieved rather than from the sample size the method prefers. Third, the risk register as written is an example of what this unit calls a register performed rather than used: running out of time names a symptom that every project shares rather than a threat this project owns. The two threats a register most needs to name are ethics delay and recruitment failure, and both are visible here. The application is deferred until the interview schedule is final, which puts a committee cycle on the critical path, and recruitment runs through home managers, who may refuse, delay or nominate convenient rather than representative staff. Each entry needs a threat, an owner and a response. Fourth, the supervision arrangement should not assume that comments help. Kluger and DeNisi (1996) meta-analysed 607 effect sizes and found an average improvement but a decrease in performance in over a third of cases, with effectiveness falling as attention moves from the task towards the self, so the agreement should say what kind of feedback is wanted and how far ahead a draft must arrive. Carless and Boud (2018) add that uptake is a student capability, so the agreement can secure the conditions for feedback but not the revision itself. Pyhalto and colleagues (2015) found that the fit between what supervisor and student believe supervision to be, across who is involved, how frequent it is and what the supervisor's task is, related to satisfaction, which is why weekly meetings alone settle nothing, and Acker, Hill and Black (1994) concluded that supervision is better described as negotiated than as managed. Finally, take a position. My advice follows Zwikael and Ahn (2010), who found that even moderate levels of risk planning reduced the negative effect of risk on project success, so a short and specific register is worth writing. The strongest objection is Bannerman's (2008), that risk management under-performed in practice and that formal project management is neither necessary nor sufficient for success, which means the register may become a document that is filed rather than used, and Manathunga (2007) sharpens the point for the agreement by arguing that the vocabulary of collegial supervision can mask the power the relationship contains. What would settle the dispute is a study that measures the specificity of registers, whether each entry names a threat, an owner and a response, against outcomes, and a comparison in which supervision terms are stated in advance in one arm and negotiated as usual in the other.

Begin with the genre. Swales (1990) shows that the introduction of a research article is a sequence of moves whose readers already know what it must do, and Punch (2006) writes for the practical reader who must judge whether the study can be run. This plan does neither job: 'much has been written' states that reading has happened rather than what claim the literature establishes, and the proposed chapter list is a container with no argument in it. Rewrite the opening to state the gap and the position, and rewrite the method sentence to name the alternatives rejected, including what observation on a ward buys over a survey of staff recollections. Second, structure the argument. Mensh and Kording (2017) instruct writers to state the central contribution in one sentence and to apply a context, content and conclusion scheme at the scale of paper and paragraph, and they extend the first four rules to other forms of communication such as grants. The test here is blunt: the student should be able to say the takeaway in one sentence, and every paragraph should carry a context that makes its content meaningful. Sword (2012), analysing more than a thousand peer-reviewed articles, found a gap between how academics describe good writing and the prose they produce, so a hedging register is a habit rather than a requirement. Third, the viva. Tinkler and Jackson (2004) rest their handbook on a four-year programme of surveys and interviews with candidates and academics and present the viva as having identifiable stages and questions, so unpredictability is the wrong premise. Bourne (2007) advises that an important talk should not be given for the first time to an audience of peers and that the take-home message should be persistent enough for a listener to recall three points a week later; the student should rehearse the reverse-result question and the recruitment-failure question with the supervisor. Fourth, dissemination. Suber (2012) treats the publication route as a choice, Penfield and colleagues (2013) find impact variously defined and assessment methods contested, and Pfeifer and Helming (2024) argue from a review of seventy impact-assessment approaches that impact can be planned for as well as measured. A plan that leaves the venue and the intended audience to 'if there is time' has postponed a decision it can make now. Finally, the contested question. The strongest objection to this advice is that genre conformity may buy readability rather than warrant: Tennant and colleagues (2017) document peer review as biased and unreliable, and Gilbert and Mulkay (1984) show that a formal register performs warrant rather than demonstrating it. What would settle it is a blind comparison in which examiners rate two versions of one proposal differing only in genre conformity, scoring warrant rather than clarity, plus a reliability study of examiner agreement; no source here reports either, so the advice remains a judgement rather than a settled finding.

A strong answer commits to one primary type and makes the rest follow from it. If the paper is an artifact contribution, the extension is the contribution and the study is its evaluation: the claim is that the extension makes a class of contrast failures visible to non-expert authors, and that its design resolves a named trade-off between coverage and false alarms. The study then supplies evidence for what the artefact makes possible, and it does not have to establish a general benefit. If the paper is empirical, the study is the contribution and the extension is the instrument: the claim is a finding about how eighteen participants located and fixed contrast errors with and without tool support, judged on the importance of the finding and the soundness of the method. Either choice is defensible. The project fails at this point only if it claims both at full strength, because then the reader cannot tell which set of criteria applies. Naming the type also fixes the scope of the paper. An artifact paper need not have produced an effect at all, so a mixed or null study does not sink it, provided the design reasoning is extracted and stated. An empirical paper built on eighteen participants must state the limits that the sample and the setting impose. The answer that earns the marks names a type, gives the matching claim, states the weaker claim about the other component, and says why the alternative type was rejected.

The limitations paragraph should separate what the study did from what the sentence claims. Twenty right-handed students using the system once in a laboratory is a narrow sample, a single session and a single setting, so the finding is about first-use performance by that group under those conditions. The order of conditions matters: if every participant used the gesture interface first, practice and fatigue are confounded with the condition, and the measured difference may be an order effect rather than an effect of the interface. Statistical significance is not the same as a large or useful difference, and Hornbaek and colleagues note that replications in HCI rarely compare effect sizes in detail, so the size of the advantage is exactly what a reader needs and what the paper does not supply. The answer to the examiner's question is that the finding does not show superiority in general: not for left-handed or older users, not after learning, and not outside the laboratory task. The stronger claim would need a broader sample and a counterbalanced or randomised order, a second session to separate learning from the interface, a task that resembles real use, and an effect size reported with a confidence interval rather than a p value on its own. A defensible wording is that participants were faster with the gesture interface on this task in a single laboratory session, which is a smaller claim and a real one.

Begin by separating the two candidate contributions, because they attract different criteria. If the tool is the primary contribution, this is an artefact claim, and Wobbrock and Kientz (2016) say artefact contributions are judged according to what kind of artefact gave rise to them, on what it makes possible and how. The CHI page makes the same demand as a question: a systems paper does not necessarily need a user study, but it does need a description detailed enough for an experienced researcher to implement something similar and a reasonable plan for validation (ACM CHI, 2027). If the trust effect is the primary contribution, the paper is an empirical claim, judged mainly on the importance of the finding and the soundness of the method, and an 18-participant study raises the evidential bar that the student must then meet. I would advise declaring the artefact as primary and the trust finding as exploratory, because the tool is what the project actually built and the study was not designed to test that effect. The unexpected finding must be labelled as emerged, not predicted. Presenting it as a hypothesis the student always had would be the failure that HARKing names; the honest form is to report it, say it emerged from the data, and mark it as requiring a study built to test it. This is also where the boundary of the claim sits. The artefact claim may say what the tool makes possible and how; it may not assert that deferring suggestions causes trust in general, because the study was small and was not designed for that contrast. The strongest objection is Fallman's (2003) point that design-oriented research and research-oriented design are different activities: a student may reply that the trust finding is the more valuable knowledge and that subordinating it to the artefact undersells the project. The reply is that the student may declare a different primary type, but must then accept its criteria and supply the method that the empirical claim owes, which an opportunistic 18-participant study does not provide.

Start with the role, because the unit argues that the role given to the object decides what the research can claim. Wensveen and Matthews (2014) differentiate roles such as the prototype as an experimental component, which stands to a hypothesis roughly as a physical test does, from the prototype as a means of inquiry, which records phenomena as an instrument would. This device was not designed to test one contrast against a control, so the experimental-component role would misdescribe it. Odom and colleagues (2016) offer the better fit: their research product extends the research prototype for inquiries into long-term relations with technology, and is characterised by four qualities, inquiry-driven, finish, fit and independent. The fact that the households kept using the device after the study is evidence of exactly the durability their account is built to address, so I would advise claiming a research product and treating the device as a means of inquiry into sustained domestic use. The second decision is documentation and presentation, because Zimmerman, Forlizzi and Evenson (2007) locate the contribution in the design reasoning rather than the object, and Pierce (2014) sorts research through design artefacts into types whose making must be made visible. The student already holds the substrate: the dated build photographs and the decision log. Bardzell, Bardzell, Dalsgaard, Gross and Halskov (2016) supply the framework for turning it into an argument, asking about the medium of documentation, its performativity, that is what recording does to the project as it proceeds, and whether the record serves both the research and the design aims. The write-up should present the reasoning that resolved the trade-offs, not just the finished device. The third decision is how general the claim is, and here the student must choose a side. I would advise following Gaver (2012), treating the contribution as provisional and contingent, and offering the work as an annotated design example within a portfolio rather than as a comprehensive theory of domestic technology, while acknowledging Redstrom's (2017) programme as the route by which a later study could stack cases like this one. What the paper may claim is what the device made visible about living with it over weeks, with the decision log as evidence. What it may not claim is a general causal effect of the design, because the households were few, self-selected and studied without a control, and the interviews were not designed to isolate one. The strongest objection is Gaver's own: the annotated artefact may be taken to carry the knowledge without the prose reasoning the unit asks for, so insisting on an extracted argument could impose a text-first standard that research through design rejects. The reply is that the annotation still has to be written, and it is the annotation, not the device, that a reader without the device can examine.

The first thing to change is the introduction, because that is where the offence described by Kerr (1998) is committed. Presenting a post hoc hypothesis as though it were a priori is the definition of HARKing, and no datum is altered by the rewrite; the paper as written invites the reader to believe that the interaction was foreseen when it was not. The interaction should stay in the paper, reported as exploratory, with the analysis that produced it named. Nosek and colleagues (2018) make the distinction that governs this: hypotheses are generated with existing observations and tested with new ones, hindsight bias makes the two hard to keep apart, and preregistration exists to mark which is which. It does not forbid the exploratory analysis; it requires the label. The second change is the method section. The prompt change is a version boundary, and the exclusion is a deviation, which is exactly the material Claesen and colleagues (2021) found being left undisclosed: of twenty-seven preregistered papers, one disclosed all its deviations and nine disclosed none, with the undisclosed ones clustering in sample size, exclusion criteria and analysis. The student's reasoning that nothing in the registration was contravened is the error the audit documents, because an unreported deviation is what turns a registration into a misleading document. Third, the registration itself should be cited and read against the paper, as Cockburn, Gutwin and Dix (2018) intend the instrument to be used. The paper should state what was registered, what changed and when, and note that the registration cannot vouch for anything it does not contain, since Yamada (2018) shows that no method currently lets reviewers or journals detect pre-registering after the results are known and Bakker and colleagues (2020) found even the structured format only reduced opportunistic flexibility rather than removing it. On the examiner's question, the honest answer is that the study was planned as one thing and has become two: a confirmatory test of the registered contrast and an exploratory finding about input method. Haven and Van Grootel (2019) support that answer, since flexibility can be tracked rather than forbidden, provided the tracking is published. The strongest objection is that a strict reading of the planned and emerged boundary would push the interaction out of the paper altogether, and that a project should not be penalised for finding more than it registered. That objection confuses labelling with prohibition. Nosek and colleagues endorse exploratory analysis, and Simonsohn, Simmons and Nelson (2020) show how much can be reported when every defensible specification is disclosed rather than one selected. The paper is not weakened by being labelled exploratory; it is weakened by being labelled confirmatory when it is not.

The single study cannot settle the claim, and the abstract should stop implying that it does. The Open Science Collaboration (2015) replicated 100 psychology studies and found replication effects about half the original magnitude, with 36 per cent of replications significant against 97 per cent of originals, so a first significant result is a first estimate rather than a fact. Camerer and colleagues (2018) found the same half-size pattern in social science, and Errington and colleagues (2021) show that a result whose protocol and data cannot be recovered cannot even be checked. The abstract should report the difference with an interval and the smallest effect the design could detect, and the results should not read p = 0.04 as proof. Loken and Gelman (2017) explain why 24 participants make the position worse: in small noisy studies, selecting on significance inflates the estimate, so the observed difference is likely to be larger than the underlying one. Scheel, Schijen and Lakens (2021) show the literature-level version, with 96 per cent positive results in standard reports against 44 per cent in Registered Reports. The examiner's objection is partly fair and partly answerable: Maxwell, Lau and Howard (2015) are right that one failed replication is not a refutation, and Klein and colleagues (2014) show some effects replicate robustly, but that cuts both ways, because it means one significant result is equally weak. The stronger claim needs a pre-registered repeat at larger scale with a defined effect size, scored on a stated replication distance as Buzbas and Devezer (2024) propose. The strongest objection is that this advice would stop a student reporting a real effect at all; the answer is that the effect can be reported as a bounded estimate without being sold as a general improvement.

The abstract claims more than the study can carry. Hornbaek and colleagues (2014) found that only 3 per cent of 891 papers across four HCI outlets attempted to replicate an earlier result, so no second study is likely to test this one and a single result bears the whole claim. The abstract should report the difference with an interval and the smallest effect the design could detect, and should name the interface, the task and the sample as conditions rather than address users in general (Hornbaek, 2013). Caine (2016) found the most common sample size at CHI 2014 was 12, so 24 participants is typical rather than sufficient, and the number has to be justified against what the design could detect. The student cannot claim that users prefer the new interface unless preference was measured, because Hornbaek and Law (2007) found perception measures largely uncorrelated with objective measures. Sonderegger and Sauer (2013) show how conditional such outcomes are: 64 participants in Switzerland and East Germany rated a high-value product in opposite directions, so the conditions change the conclusion. To support a general claim the follow-up must be a pre-registered repeat that deliberately changes the interface, the task or the population and reports the size of that change (Wilson et al., 2011; Iarygina et al., 2026). The strongest objection is that this advice discourages any general claim at all; the answer is that a bounded claim is still a claim, framed so that a reader knows exactly what was tested and what was not.

The sentence reports a threshold crossing and nothing else, so it answers a question the reader was not asking. It should be replaced by the mean difference, its confidence interval and the effect size with the measure named (Cumming, 2014; Lakens, 2013), together with a statement of the smallest effect the 20-participant design could detect, computed as Cohen (1992) describes. The p value may stay as one number among these, because the field expects to see it, but it must not carry the verdict. Button and colleagues (2013) estimate median power in the neurosciences at between about 8 and 31 per cent, so a study of 20 almost certainly cannot detect a modest effect, and its significant results are the least trustworthy part of its output. The student may adopt any reform position but must state and defend it: keeping 0.05 by convention is what Lakens and colleagues (2018) reject; a 0.005 default is what Benjamin and colleagues (2018) propose and what Ioannidis (2018) calls a temporizing measure; abandoning the threshold follows McShane and colleagues (2019). The strongest objection is that this advice makes a small study unusable; the reply is that the study is already weak, and clearer reporting only makes its limits visible.

The claim overreaches three times. Engagement is a construct the project has not defined or placed in a network of predicted relations, so the measure cannot yet be argued to be a measure of it (Cronbach and Meehl, 1955). The five-item questionnaire written for the study carries no validity evidence beyond its own internal consistency, and Flake, Pek and Hehman (2017) found that coefficient alpha is often the only psychometric evidence a paper offers for a scale, which is exactly the position here. The sample is 18 students from one seminar group on one task in one sitting. The replacement sentence should report the score, the measure and the conditions, and say what the study did not show: preference is not performance (Nielsen and Levy, 1994), and a single session cannot speak to sustained engagement. Each threat costs something nameable. Sampling and no randomisation threaten internal validity, so the claim of cause is unsupported. The ad hoc measure threatens construct validity, so the word engagement may not travel from the items to the concept. The narrow sample and setting threaten external validity, so the result is a working hypothesis about these conditions, in Cronbach's (1975) sense, not a property of the design. The evidence the project has, that item responses moved together and varied with the condition, addresses Borsboom and colleagues' (2004) causal account only weakly, and Messick's (1995) inference and consequence account not at all, since no consequence of the tool's use was examined. Under Kane (2013) each inference from response to score to construct to population needs stating. The strongest objection is that this reporting standard makes small student projects unclaimable, and the reply is that the claim is already weak and the wording only makes the weakness visible.

The abstract fails the function Zobel (2014) gives it, which is to let a reader judge relevance in fifty to two hundred words, so it should state the question, the central contribution and the conclusion rather than the procedure. On Mensh and Kording (2017), the paper should carry one central contribution, named in the title, and each unit of text should move from context through content to conclusion; the student should be able to say what a reader who was not present would extract from the structure alone. The related work is the larger problem. 'No one has done exactly this' is a claim to novelty without a niche, and Swales (1990) requires the section to establish a territory, indicate a gap and occupy it. Webster and Watson (2002) require the twelve summaries to become concept-centric, organised by idea with a concept matrix, since an author-centric list does not synthesise. Pautasso (2013) adds that the topic must be a well-defined issue, and Boote and Beile (2005) show that justified inclusion and exclusion criteria are exactly what such sections omit. The results order should follow Whitesides (2004), who asks for importance rather than chronology because the reader cares what the results are, not how they were reached. Whether these rules are general is settled only by evidence: the same content in two arrangements, judged by readers who do not know the study, or structural features coded against review outcomes (Hyland, 2004). The strongest objection is that this makes the paper legible to a reviewer at the cost of the intellectual history the student actually did, and the reply is that the history belongs in the log the paper was made from, not in the argument made to the reader.

The claim of availability is not the claim the student needed. The ACM (2020) separates the availability badge, a property of the deposit, from the evaluated badge, which requires a package that is documented, consistent, complete and exercisable, and the missing version and broken path are precisely failures of exercisability. The package should therefore contain the analysis script, the instrument, the codebook, the derived data and a readme naming the software and package versions and the environment, following Eglen and colleagues (2017), for whom licensing, documentation and archiving are what make a package usable. The test is the one Collberg and Proebsting (2016) applied: build and run it on a clean machine, and regenerate every number the paper reports, since Stodden, Seiler and Ma (2018) found that artefacts were obtained for 44 per cent of a sample while only 26 per cent reproduced. Trisovic and colleagues (2022) locate most residual failures in missing dependencies, which the readme and a pinned environment address. Because the project studied people, some material may be unshareable, and the paper should name what is withheld and why, as Bietz and colleagues (2015) require, rather than implying that everything is present. The strongest objection is that re-execution depends on the tester: Collberg and Proebsting (2016) found that more effort raised the build rate from 32.3 to 54.0 per cent, and Trisovic and colleagues (2022) raised execution after automatic cleaning, so what runs is partly a function of the agent and environment, not a fixed property of the package. The reply is that a package documented and exercised by its author shifts that burden from the examiner to the person who wrote the code.

The answer to the examiner is a version boundary, not an apology. The student should say what changed, on what date, and why, and then say what the change cost the comparison: the two halves used different instruments, so any difference between them is confounded with the instrument revision. The record that would have made this answerable is the one the unit describes, a dated decision log with the reason for each decision, the instrument file as it stood at each version, and separate files for observation and inference, which is the practice the unit treats as the taught part of a supervised project. Kiley and Mullins (2004) found that less experienced examiners were unsure of the boundary conditions that separate a very good thesis from a poor one, so the student should state the boundary before it is probed: the claim covers the later sessions, or it covers the combination only as a descriptive account, and the study does not show that either interface is better in general. That is the move the unit's thesis predicts will be rewarded, because the criteria examiners named include argumentative rigour, expected to be evident throughout the work, and a coherent whole, and neither is settled by reciting results (Adunyarittigun et al., 2026; Golding et al., 2014). The counter-argument is that if the criteria are as stable as Bourke and Holbrook (2013) found, with 427 examiners rating twelve indicators very similarly, then the written thesis decides the outcome and the viva merely confirms it, so effort spent on the record would be better spent on the thesis. The reply is that coherence and argumentative rigour are tested in speech, and that the record is the cheapest way to make the reasoning behind an inconsistency visible.

The request is a request, not a question, so the first move is to name the judgement. 'How air quality has changed' usually means a trend over time, and possibly a comparison with a guideline or a previous period, so I would name the judgement as a trend against a reference rather than a bare sequence of values. That choice fixes the encoding: a line chart on a time axis, with position carrying the magnitude and a labelled reference line for the legal limit, because position on a common scale supports the most accurate comparisons and the reference turns an abstract series into a judgement a resident can hold (Cleveland and McGill, 1984; Munzner, 2014). I would not use a colour-coded map as the primary figure, because a map encodes geography alongside the value and invites a reading of where rather than whether, and residents asking how it has changed are asking about time. For the medium, the figure must work on a phone in daylight and in greyscale print, so I would avoid a green-to-red ramp, carry the categories with labels and shape as well as hue, and check the palette under colour-vision simulation (Crameri, Shephard and Heron, 2020). I would state the measure and its averaging period, because a daily maximum and a weekly mean tell different stories. The one thing I would test is comprehension of the trend and the reference, not preference: a short task in which a reader says whether the last month is better or worse than the limit, and why. If they cannot answer, the encoding or the annotation is wrong, and no amount of colour will fix it.

A single measure would hide the question, so I would state three targets and choose methods for each. The first is comprehension: can a reader answer a small set of realistic questions, such as which area is worst and whether the difference is meaningful? A task-based comprehension test is repeatable, but it can only report on the searches the test taught, so I would design tasks from the questions the public actually asks rather than from the interface's features (Saraiya, North and Duca, 2005). The second is appropriate reliance: does the reader treat small differences as small and large ones as large? Because the data is uncertain, I would include a judgement where the difference is within the interval and check whether the display lets a reader say so, since the uncertainty evaluation literature shows that accuracy on an abstract task and good decisions can diverge (Hullman, Qiao, Correll, Kale and Kay, 2019). The third is exploration: did the reader find their own question, which is the purpose of an interactive view, or did they stop at the first screen? I would measure this with simple interaction logs plus a short think-aloud, and treat the logs as a proxy rather than as evidence of insight. I would also check that the narrative framing did not steer readers to the author's conclusion: I would show two versions with the same data and different annotation and compare the answers (Segel and Heer, 2010). The decision rule is set in advance: the visualisation works if readers can answer the comprehension tasks and correctly treat the uncertain comparisons as uncertain, and it fails if they are confident about differences the data does not support, whatever they say they liked.

The brief names objects, not a question, and the first job is to convert it. It asks for maps and a risk score, which are forms; it does not say what judgement the reader has to make. Mackinlay (1986) shows why the conversion cannot be postponed: effectiveness criteria rank designs, but expressiveness criteria come first and ask whether a graphical language can express the desired information at all, so the information has to be named before a form can be defended. I would write down one question and one reader, for example: 'For which districts has the rate of a named condition risen faster than the regional trend over the past three years, for a board member who will decide where to place a two-year intervention?' Munzner's (2009) nested model supplies the reason for the discipline: an effective design solution at the encoding level cannot repair a wrong abstraction, because an upstream error cascades to every downstream level. The public and the board are also not one reader, and a chart that answers neither is worse than one that answers one of them well. I would then decompose the question with Amar, Eagan and Stasko's (2005) ten low-level tasks: determine range to fix the period, correlate to compare a district's trend with the regional one, find extremum to identify the worst case, characterise distribution to show spread across districts, and find anomalies to flag the district whose rise is not explained by its size. That decomposition decides the marks: a small multiple of trend lines with the regional line as reference, not a choropleth, because Shneiderman's (1996) taxonomy keeps task and data on separate axes: a map answers where a value sits, and the question I named is a comparison of trends over time. Brehmer and Munzner (2013) make the same point as a grammar, since the why has to be settled before the how, and Shneiderman (1996) adds that a reader moves between tasks within a session, so the view should support overview, zoom and details on demand rather than a single static answer. For evaluation I would run one scenario, not a comprehension quiz: Lam, Bertini, Isenberg, Plaisant and Carpendale (2012) derive seven, and this commission is a case of evaluating communication through visualisation, so the target is what board members take away and act on, not how fast they read a bar. North's (2006) argument applies directly, because a benchmark task would test the question I supplied rather than the question the board brought, and an insight-based method records what the reader actually notices. Once the first task is finished the dashboard should still carry the region-wide trend, the count behind each rate, and the interval around it, because the reader's next question is whether the rise is real and what is driving it. The counter-argument is Sedlmair, Meyer and Munzner's (2012) task clarity axis: where the task is genuinely fuzzy, insisting on one named question can foreclose the finding that matters, and their framework places winnow before discover precisely so that a study is chosen only when the problem warrants it. My answer is that the axis decides the depth of the work, not whether the question is written down. A fuzzy task is clarified by naming a first question and stating the assumptions behind it, which is what Munzner recommends when she asks authors to declare the layers above their own; what it rules out is arriving at week four with a choropleth and no statement of what it was meant to answer.

The brief asks for consistency, and the first move is to say what consistency means. I would hold three things constant: the question, the ordering of the twelve regions, and the shared scale, because Wickham's (2010) grammar shows that locally scaled facets are not necessarily comparable across facets, so any surface that forces local scales has already broken consistency in the sense that matters. The form itself I would choose per surface. For print, a set of horizontal dot plots on a common horizontal scale, because Cleveland and McGill (1984) rank position along a common scale as the most accurate elementary task, and Talbot, Setlur and Anand (2014) show that separating the marks raises error, with an estimated separation effect of 0.81 percentage points, so adjacent rows on one axis are worth the space. For the slide, the same dot plot with fewer regions shown at once and the annotation carrying the headline, because a projector is read at distance and the interval is the part most likely to be lost. For the phone, I would test the chart at a height near the floor Heer and Bostock (2010) measured rather than shrinking the print figure, since they found significantly more error at 40 pixels and little benefit above 80 on a 0 to 100 scale, and I would drop the least important regions rather than the interval. For the wall, a scatter of estimate against region with the interval shown, because Ball and North (2005) show that a wall changes the reader's behaviour rather than only the resolution, and Chittaro's (2006) list of mobile constraints is the reminder that a surface brings input, distance and context with it. Before testing, I would predict each change from the channel ranking: converting position to angle or area should cost more than converting one position encoding to another, and shrinking below the measured floor should cost accuracy at a rate the bar chart experiments already bracket. To check the decision I would run one measurement, a value-estimation task on all four surfaces with the same data and the same question, recording error, time and navigation, because van Wijk's (2005) cost model means accuracy alone cannot decide, and Zacks and Tversky (1999) warn that changing bars into lines changes what the reader takes the chart to be saying. The strongest counter-argument is that four bespoke forms multiply development and per-user cost, and van Wijk's (2005) accounting makes the value of a method depend on how widely it is used and how little it costs to use routinely. My answer is that the shared scale and the shared ordering are what make the four versions one figure rather than four, and that the per-surface change is confined to the encoding layer, so the development cost is a parameter change rather than four designs.

Because the variable is ordered and single-directional, the scheme is sequential rather than qualitative, and the match between the nature of the data and the class of scheme is the first decision (Harrower and Brewer, 2003). I would build the palette by ramping in luminance, adding chroma and hue only to raise contrast between neighbouring steps, which is the sequential construction Zeileis, Hornik and Murrell (2009) give, because luminance survives both greyscale printing and colour-vision deficiency: it stays visible even to a monochromat, where a red-green hue contrast does not. What must stay constant is the classification, the number of data classes and their boundaries, so a reader comparing two printouts is comparing the same bins. What may change is the number of classes actually shown: on a phone I would merge the lowest bins rather than shrink the marks, since the smallest differences will not resolve. Before testing I would predict, from Ware (1988), that the greyscale and the deficiency conditions are the two most at risk, because his result is that a grey scale is weak at conveying value while a scheme built on hue loses form information it carried; and from Olson and Brewer (1997), that a deficiency-safe palette restores accuracy but may leave a reaction-time cost. The check I would run is a greyscale and deficiency simulation plus a small reading task on the projected and printed versions, comparing a luminance-ramped scheme against a hue-only scheme on the same variable. The strongest counter-argument is Szafir's (2018): colour difference is not perceived identically across mark types, so a palette validated on large fields may fail on the small elongated marks a rate map uses. I would answer it by running the check on the actual marks the figure will use rather than on colour swatches.

The board's decision is about the size of an effect on individual patients, not about how precisely a mean has been estimated, so I would show outcome uncertainty rather than inferential uncertainty. Hofman, Goldstein and Hullman (2020) found that readers shown 95 per cent confidence intervals overestimated the treatment effect and its probability of superiority and understated the variation in individual outcomes, while readers shown prediction intervals were closer to the normatively correct answer and animated hypothetical outcome plots did best of all. I would therefore draw the estimated change with a prediction interval, or animate a set of hypothetical outcome draws, because Hullman, Resnick and Adar (2015) found the animated draws supported markedly more accurate judgements about more than one quantity. I would avoid a bar chart with error bars, since Correll and Gleicher (2014) found it carries a within-the-bar bias and makes readers predict larger effects with more confidence than the data support, and I would not rely on axis rescaling, which Hofman and colleagues found helped less than prediction intervals or animated draws. Before testing I would predict, from Correll and Gleicher (2014), that readers overestimate the effect when it is drawn as a bar with an interval, and from Cumming, Fidler and Vaux (2007), that they misjudge significance from overlap, so the risk is a reader treating a modest effect as decisive. The check I would run is a short reading task on the actual figure, asking board-style readers to state whether the service helps an individual patient and how confident they are, and comparing the prediction-interval version against a bar-with-error-bars version. The strongest counter-argument is Spiegelhalter's (2017) point that no single representation suits every member of an audience and that numbers are needed for detail, so a graphic aimed at gist may leave some readers without the precise estimate they want. I would answer it by showing the prediction interval in the graphic and stating the estimate and its interval in words and numbers beside it, since Spiegelhalter recommends illuminating graphics with words and numbers and using multiple formats.

The board's wanted conclusion is a reading, not a fact, so the first decision is to disclose it rather than let the figure imply it. I would include all twelve regions, the population or case count behind each rate, and the two-year series for the three named regions, because Pandey, Manivannan, Nov, Satterthwaite and Bertini (2014) found that a figure moves the attitudes of readers who hold no strong prior view, and a press release is aimed exactly at those readers. Following the visualisation rhetoric framework of Hullman and Diakopoulos (2011), I would treat the annotation, the selection and the order as the places where framing enters, and I would state in the figure's own text the reading it supports and what would count against it. I would annotate the three regions with their population and the prior-year base, and I would mark the small-numbers caveat inside the figure rather than in a note, because Shah and Hoeffner (2002) identify the reader's expectations about the content as one of the three influences on comprehension, and the annotation is what supplies that expectation. In sequence I would open with the twelve-region overview, then the three changes with their denominators, and I would use parallel structure for the three panels, because Hullman, Drucker, Riche, Lee, Fisher and Adar (2013) found memory and rating benefits when a sequence uses parallelism as a structural device. I would avoid an animated transition entirely and would not stagger any motion, since Robertson, Fernandez, Fisher, Lee and Stasko (2008) found animation least effective for analysis and most error-prone in presentation, and Chevalier, Dragicevic and Franconeri (2014) found the popular staggering convention did not help tracking. Before release I would predict that readers take the three increases as evidence about the whole service and ignore the small denominators, which is the pattern Pandey and colleagues measured in attitude change. The one check I would run is a short reading task with readers of the target audience asking two things: what they now believe about the service nationally, and what they think could count against that belief. The strongest counter-argument is that annotating the denominators and the counter-case weakens the charity's case, and that rhetorical restraint is a luxury an underfunded service cannot afford, which is the position Kosara and Mackinlay (2013) imply when they treat storytelling as central to communication. I would answer it by noting that the funding argument depends on a lasting change in attitude rather than an immediate one, that Pandey and colleagues found readers who start opposed are moved by tables rather than charts, and that a figure which raises agreement while erasing the counter-case is the distortion the unit defines, not the communication.

I would start from the reader's intent rather than from the controls, because Yi, Kang, Stasko and Jacko (2007) built their seven categories around what a reader is trying to do, and Lam (2008) shows that decision cost, the step of forming a goal, is charged before any control is touched. So the page opens with one default view: 1,200 routes as a ranked distribution of on-time percentage, with the 40-journey median stated, which is a question a reader can answer without acting. I would provide three controls only, each mapping to one intent: a filter by region, a search by route number, and a toggle between route-level values and the distribution. I would refuse a free-form expression field, linked multi-view brushing and any animated transition, because Heer and Shneiderman (2012) place coordination and derivation far along the analytic dialogue, and a reader with under a minute will not pay their cost. I would also refuse, on the press office's instruction, to show only the best-performing regions: the framing the office wants is a claim about selection, and omitting the rest would enter the page through that omission in the way Hullman (2020) documents for uncertainty. For readers who never interact, the page must answer the press office's question in static form before the controls appear, because Boy, Detienne and Fekete (2015) found in three field experiments that adding an introductory story to an exploratory view did not increase exploration, so I cannot assume an on-ramp will convert a non-interacting reader. Muller and colleagues (2009) reported the denominator of their display study, 91 users at 11 displays, and I would do the same: log how many readers see the controls at all, and treat that number as a finding. Before release I would predict two errors, both drawn from the sources rather than from intuition. The first is that readers take the map of worst routes as evidence about the network, when a route with 40 journeys produces unstable rates by construction. The second is that readers without the literacy to operate the toggle never see the distribution and read the ranking as the whole picture, which is the entry requirement that Boy, Rensink, Bertini and Fekete (2014) and Lee, Kim and Kwon (2017) make measurable rather than assumed. The one check I would run is a short task-based session with readers recruited from the news traffic rather than from the authority's staff, in which each participant is asked for the question they arrived with, then for their answer, then for the strongest argument against it, with the time taken recorded; Kim and Heer (2018) show that encoding effectiveness depends on the task and the data distribution, so the check must test my readers' tasks and not my own. The strongest counter-argument is that interaction is exactly what a 1,200-route dataset needs, because no single static figure can carry that many cases, and that refusing controls privileges the press office's summary over the resident who wants their own street. That argument is right about the data and is the reason the default view must show the whole distribution rather than an average. It is wrong only if it assumes the reader will interact; the check above measures whether they do, and if they do not, the static distribution is doing the work and the controls are decoration.

I would start from the denominator, because with four cases in the smallest district any rate is unstable by construction, and Gelman and Price (1999) show that plotting observed rates places the extremes in the thinly populated areas while adjusting them moves the extremes to the well-sampled ones, so a bare rate map tells the reader about population size rather than about illness. I would therefore show no rate without its count, and I would not rank districts without that count beside them. I would use an age and sex adjusted rate, following Beyer, Tiwari and Rushton (2012), whose first properties ask for a controlled population basis, continuous display and maximum geographic detail, and I would follow Clayton and Kaldor (1987) in using empirical Bayes estimation so that small districts are pulled toward their neighbours, while making clear that this shrinkage depends on sample size and is a design decision rather than a correction I can assume. For classing I would use an equal-count method with a diverging, colour-vision-safe scheme rather than the data dependent optimum, because Brewer and Pickle (2002) found quantiles and minimum boundary error best for general map-reading tasks and the heavily optimised natural breaks less than seventy per cent as accurate for those tasks. I would refuse the ranking the press office asked for, and refuse any map that shades a raw rate without its count, because such a map invites the ecological fallacy Robinson (1950) demonstrated when the sign of a relationship reversed between the state and individual levels. For readers who never interact I would put a three-item static summary above the map, naming the denominator rule, the adjustment and the uncertainty, since the field evidence is that adding an on-ramp does not convert non-interacting readers. Before release I would predict two errors: that readers take the named districts as bad places rather than as small ones, which is the consequence of the denominator, and that they read the smoothed surface as measured fact when the shrinkage has borrowed strength from neighbours. The one check I would run is a short task-based session with readers resembling the audience, asking them to state the rate and its count for two districts, one large and one small, and then to give the strongest argument against the conclusion they drew, with the proportion who can supply it recorded. The strongest counter-argument is that omitting the ranking hides the authority's real finding and that smoothing may bury a genuine local cluster, since Anselin's (1995) local statistics exist precisely to detect local pockets of nonstationarity. That argument is right that local clusters matter, which is why the page should carry the local statistic and the count in the tooltip, and it is wrong only if it treats a small district's high rate as evidence of a cluster rather than as the expected behaviour of a small denominator. The counter-argument changes what the page discloses, not whether the denominator is shown.

I would keep the 25 per cent figure off the page as the headline, because it is a relative risk reduction and Covey (2007) found that relative risk formats produce more favourable evaluations of treatments than absolute risk or number needed to treat formats, which means the figure moves the reader while telling them little about what will happen to them. Following Gigerenzer and colleagues (2007), I would present the same information as natural frequencies with an explicit denominator, that about 3 people in 1,000 die of bowel cancer without screening and about 2 in 1,000 die with it, so the reader can see that the 25 per cent reduction is one fewer death in 1,000. I would state the reference class and the time frame, as Trevena and colleagues (2013) require, and avoid the 1 in x form and any change of denominator between sentences. For the visual, I would use an icon array of 1,000 figures showing the whole population at risk rather than only those affected, because Garcia-Retamero and Galesic (2010) found the largest accuracy gains when the aid depicted the entire population at risk, and I would accept that a larger array changes how serious the risk feels, since Galesic and colleagues (2009) found arrays of 1,000 icons raised perceived seriousness relative to arrays of 100. I would refuse the 25 per cent headline and refuse any figure without its denominator, because Reyna and Brainerd (2008) describe denominator neglect as the failure this design is there to prevent, and I would refuse decoration that increases liking without improving accuracy, which Ancker and colleagues (2006) found were different properties. Before release I would predict that readers take the screening as preventing most bowel cancer deaths rather than one in 1,000, the error Covey's wording finding and Reyna and Brainerd's denominator neglect both predict. The test I would run is a comprehension check that asks readers to recover both the absolute number helped and the denominator and to state the case against taking the test, not a rating of how clear the page looked. The strongest counter-argument is that a bare icon array and a frequency sentence may be less persuasive to people who avoid numbers, so fewer may take the test. I would answer it by noting that Lipkus (2007) found few formats for conveying risk had been tested against one another on the same outcomes, so persuasion is an open question rather than an established benefit of opacity, and that the programme's aim of informed uptake is not served by a figure that inflates the benefit.

The first thing I would change is not the panel but its attachment. The director has described an instrument that triggers an explanation, which is the governance regime Bevan and Hood (2006) examined in the English public health service, where targets were combined with what they called an element of terror, and which Ridgway (1956) warned would make the measure a definition of what matters rather than a report on it. I would ask for the panel to run for a first period for information only, with no ward asked to explain itself, so that the baseline behaviour of the measure can be seen before any incentive is attached. That is the condition under which the panel reports the ward rather than the ward's response to the panel. For the tiles themselves I would follow the separation Bach and colleagues (2022) drew between design goals and dashboard genres, and build an analytical panel for the board's decision rather than a narrative one for the meeting, because the four measures serve comparisons across wards and over time rather than a single story. Each tile would be matched to a named task, following Sarikaya and Gleicher (2018): the four-hour percentage as a position on a common scale with its denominator, incidents as counts with the small-number instability shown rather than hidden, length of stay as a distribution with its interval, and sickness rate as a trend. I would refuse the composite score the director did not ask for but will inevitably request, because Bouckaert and Balk (1991) and Ridgway (1956) both document what happens when value judgements are buried inside a single authoritative index, and I would refuse any tile that shows a rate without the count behind it. Before release I would predict that the four-hour percentage improves while patients wait longer somewhere the panel does not look, because that is the substitution Bevan and Hood (2006) documented when a target becomes the goal. The test I would run is an audit of the raw records behind the four-hour measure in two wards, before and after the target is attached, to see whether the improvement is in the care or in the recording. The strongest counter-argument is Brynjolfsson, Hitt and Kim (2011), who found firms emphasising data-driven decision making had output and productivity five to six per cent higher, which suggests that attaching consequences to measures is how organisations get the benefit at all, and that an information-only panel will be ignored. My answer is that their result is firm-level and about a management posture, not about a panel, so it cannot tell us what happens to a ward measured monthly; and the way to test the counter-argument is not to argue about it but to run the two regimes against each other, as Dowding and colleagues (2015) asked when their review found the evidence too thin for confident guidelines.

I would begin by refusing the premise that there is one job the piece does, because the unit's whole point is that comprehension, insight, confidence and decision quality are different targets and a study of one says little about the others. The lead has named comprehension, and North (2006) is the reason I would not simply accept it: if the purpose of visualisation is insight, then a multiple-choice quiz on readings that were already explained measures the search we taught, not whether a resident learned anything about their own neighbourhood. So I would keep the comprehension quiz, because it is repeatable, but I would add an open-ended insight protocol of the kind Saraiya, North and Duca (2005) built, in which readers talk through the map and I count and type the observations they reach, accepting that it is labour-intensive and hard to standardise. That addition buys me the discovery-style finding a quiz cannot record, at a named cost. Knowingly, I would give up decision quality: twenty colleagues cannot supply a decision outcome, and Plaisant (2004) and Stasko (2014) ask for the value a tool delivers in real use, including confidence and adoption, which a pre-publication session cannot show. I would say so in the write-up rather than imply the quiz covered it. The bias I would predict is an order or framing effect from the map's default view, because Dimara, Franconeri, Plaisant, Bezerianos and Dragicevic (2020) show that biases in visualisation are systematic and task-linked, and a seasonal reading invites a rank that depends on the month shown. One method I would add beyond the quiz is an annotated pre-registered list of the map's known weaknesses, scored blind against a heuristic evaluation, following Zuk and colleagues (2006), who found that heuristic results depend on the heuristics and the evaluators chosen and that visualisation-specific heuristics catch problems usability heuristics miss. The strongest counter-argument is Plaisant's and Stasko's: that a comprehension and insight study on colleagues tells us nothing about whether the published piece changes what residents do, and the effort would be better spent tracking adoption. I would answer it by agreeing on the target and disagreeing on the timing: adoption data arrive only after publication, so the pre-publication study should be scoped to what it can honestly claim, and its limits stated.