Diaries and experience sampling: capturing life as it happens

Every method so far asks a participant to produce an account after the fact. This sixth of the ten units examines the methods that refuse the delay. The diary study asks people to record events as they occur. Experience sampling prompts them at set or random moments. Recording close to the event buys freedom from hindsight. The price is paid twice. The method imposes a burden and missing entries, and the measurement changes the behaviour it measures.

A diary is not an interview with better recall. It is an instrument carried by the person being measured.

Hindsight is a measurement error, not an inconvenience

Shiffman, Stone and Hufford (2008) reviewed ecological momentary assessment. They argued that global retrospective self-report taken at clinic visits is limited by recall bias and cannot describe how behaviour changes across contexts. Their alternative samples behaviour in real time in the participant's own environment. It minimises recall bias and studies processes in the world.

Redelmeier and Kahneman (1996) recorded pain in real time from 154 colonoscopy and 133 lithotripsy patients. They compared it with their later ratings of total pain. The ratings were predicted by the peak intensity and by the pain of the final three minutes. Duration did not make the procedure remembered as worse. Memory of an episode is organised around its worst moment and its ending.

Robinson and Clore (2002) supplied the mechanism. Their accessibility model separates an emotion, which is episodic and contextual, from a belief about emotion, which is semantic and decontextualised. It predicts that people not currently experiencing a state report it by inference. Asked in December how stressful the summer was, a person answers from what they believe such a summer is like rather than from the episode. Stone and colleagues (1998) showed the same mechanism on behaviour. Coping reports made close in time and reports made within two days afterwards corresponded poorly, with cognitive coping underreported and behavioural coping overreported. A retrospective question does not collect a noisy version of the momentary answer. The two instruments measure different things.

A diary trades hindsight for the participant's effort

Bolger, Davis and Rafaeli (2003) reviewed diary methods. They set out their design space as three choices: the interval between reports, the nature of the response required and the prompting technology. The choice, they argue, decides which question can be answered at all. The method trades participant burden against reliance on memory. Burden is not shared out evenly, and a participant with a caring role or a long shift has less room to answer a prompt than one with spare time. They mark the direction of travel, towards electronic collection and collateral measures such as ambulatory heart rate.

The psychometric case came first. Csikszentmihalyi and Larson (1987) reported short-term and long-term reliability for experience sampling. Participants carried a pager and reported their state at signalled moments. Validity evidence was drawn from correlations with physiological measures, tests and behavioural indices. That momentary self-report describes the frequency and patterning of everyday experience is what makes the method usable.

None of it is free. Stone and Shiffman (2002) proposed reporting guidelines because the momentary papers they were reading could not be assessed. They listed seven things a report must state, among them the sampling rationale, the acquisition interface and the rate of compliance. A study that omits the compliance rate has published a measure whose quality is unknown.

The instrument is inside the finding

The medium is not neutral, and the clearest evidence is an act of verification. Stone, Shiffman, Schwartz, Broderick and Hufford (2003) randomised chronic pain patients to paper or electronic diaries recording symptoms three times a day for three weeks. A hidden photosensor recorded when the booklets were opened. Patients reported compliance of about ninety per cent. The sensor showed eleven per cent, the binder went unopened on about a third of those days, and the entries had been backfilled. In the electronic condition, where each entry was time-stamped, verified compliance was ninety-four per cent. The gain is uneven, because the electronic arm assumes a charged device, a data plan and a hand free to answer it. A paper diary reaches the participant who has none of those, and gives up the timestamp that makes compliance checkable.

Carter and Mankoff (2005) ran three diary studies in which participants captured experience through photographs, audio, location information and artefacts. They found the media differed in what they prompted people to notice. The choice between a camera and a page is therefore a choice about what the study can see. Ho and Intille (2005) delivered mobile messages at detected activity transitions or at random times, and found those at transitions better received. The burden of a prompt depends on when it arrives. Czerwinski, Horvitz and Wilhite (2004) call the influence of journaling on the events journaled a Heisenberg-style challenge.

A week of entries sees what an interview cannot count

Czerwinski, Horvitz and Wilhite (2004) ran a week-long diary study with eleven information workers. They recorded each task, when they started it, how hard it was to resume and how many interruptions it absorbed. Participants reported about fifty task shifts over the week and 0.7 interruptions per task, close to one for every task begun. The finding is not that interruption is frequent, which was known, but which tasks suffer. Returned-to projects were longer, absorbed more interruptions, and were poorly supported when resumed.

An interview would have produced an estimate organised around the memorable interruption rather than the count. That is the construction Redelmeier and Kahneman (1996) and Robinson and Clore (2002) predict. The diary gets the distribution. The interview gets a story about it.

Contested question: must the record be made in the moment?

The dispute is not whether momentary reporting is accurate for its purpose, which is largely settled, but whether it is necessary. Kahneman, Krueger, Schkade, Schwarz and Stone (2004) introduced the day reconstruction method. Respondents rebuild yesterday as a sequence of rated episodes, a procedure designed to reduce recall bias. Applied to 909 employed women, the reconstructed accounts showed close correspondence with established experience sampling results at far lower burden. If careful reconstruction reproduces the momentary result, the claim that only momentary assessment will do loses its force.

The opposing position rests on the error being directional rather than removable. Redelmeier and Kahneman (1996) showed that retrospective totals track the peak and the ending, not the duration. Stone and colleagues (1998) showed that recall reorganises coping around the outcome. Shiffman, Stone and Hufford (2008) present momentary assessment because recall bias is not a nuisance to be minimised with better questions. On this view reconstruction removes the error only where episode boundaries are salient. It does not remove it where the decision turns on frequency, duration or events the respondent would rather not report.

What would settle it is a head-to-head test rather than an appeal to plausibility. The same construct would be measured in the same people over the same period. It would be measured by momentary sampling and by a reconstruction instrument, with the momentary record as criterion. Kahneman and colleagues (2004) report agreement on aggregate time use and affect, not on the event-level counts this unit cares about.

A second dispute sits inside the method. Jones and colleagues (2019) pooled 126 ecological momentary assessment studies covering 19,431 participants. They found mean compliance of about seventy-five per cent, below the rate the field recommends and unrelated to prompt frequency or assessment length. van Berkel, Ferreira and Kostakos (2017) found the sampling configuration reported inconsistently. This is settled procedurally. Report verified rather than self-reported compliance, in the terms Stone and Shiffman (2002) set out. Timestamp entries as Stone and colleagues (2003) did, so that unanswered prompts become a reported property of the dataset.

The evidence on each side has a different reach. Redelmeier and Kahneman recorded pain in real time from hundreds of patients, then measured the retrospective error directly. Kahneman and colleagues set reconstructed accounts against established sampling results for a large sample of employed women. The reconstruction case is stronger for aggregate time use and affect, and weaker for the event-level counts this unit cares about. While the dispute is open, choose the instrument by the quantity. Use momentary sampling for frequency, duration and events the respondent would rather not report. The strongest objection is that a careful reconstruction can reproduce the momentary result at far lower burden.

Where the argument is heading

The direction is that machines increasingly make the record alongside people. Bolger, Davis and Rafaeli (2003) already pointed to electronic collection and collateral measures. van Berkel, Ferreira and Kostakos (2017) surveyed what mobile devices have done to experience sampling and which design parameters still need investigation. Once a phone logs when it was used, the sample no longer depends on the participant remembering to answer. That does not dissolve the unit's problem, because sensors measure behaviour and not experience. The likely divider is reporting. The standards Stone and Shiffman (2002) set out are becoming the condition on which a momentary study can be assessed. Stone and colleagues (2003) are the reason the field no longer takes a compliance figure on trust.

The sources this unit rests on

• Shiffman, S., Stone, A. A. and Hufford, M. R. (2008). Ecological Momentary Assessment. Annual Review of Clinical Psychology. doi:10.1146/annurev.clinpsy.3.022806.091415 Reviews ecological momentary assessment in clinical psychology and argues that global retrospective reports are limited by recall bias. It presents real-time sampling in natural environments as the way to minimise that bias and study behaviour in context.
• Redelmeier, D. A. and Kahneman, D. (1996). Patients' memories of painful medical treatments. Pain. doi:10.1016/0304-3959(96)02994-6 Recorded pain in real time from 154 colonoscopy and 133 lithotripsy patients and compared it with their later ratings. The ratings tracked the peak and the final three minutes rather than the duration, so hindsight errs in a knowable direction.
• Robinson, M. D. and Clore, G. L. (2002). Belief and feeling: Evidence for an accessibility model of emotional self-report. Psychological Bulletin. doi:10.1037/0033-2909.128.6.934 Proposes an accessibility model that separates an episodic emotion from a semantic belief about emotion. It predicts that people not currently experiencing a state report it by inference, the mechanism behind the recall error.
• Stone, A. A., Schwartz, J. E., Neale, J. M., Shiffman, S., Marco, C. A., Hickcox, M., Paty, J., Porter, L. S. and Cruise, L. J. (1998). A comparison of coping assessed by ecological momentary assessment and retrospective recall. Journal of Personality and Social Psychology. doi:10.1037/0022-3514.74.6.1670 Compared coping reports made close in time on a hand-held computer with reports made retrospectively. Correspondence was poor, with cognitive coping underreported and behavioural coping overreported, showing that recall reorganises rather than retrieves.
• Bolger, N., Davis, A. and Rafaeli, E. (2003). Diary Methods: Capturing Life as it is Lived. Annual Review of Psychology. doi:10.1146/annurev.psych.54.101601.145030 Reviews diary methods and sets out the design space as the interval between reports, the nature of the response and the prompting technology. The choice, it argues, decides which question can be answered and trades burden against reliance on memory.
• Csikszentmihalyi, M. and Larson, R. (1987). Validity and reliability of the experience-sampling method. Journal of Nervous and Mental Disease 175(9), 526-536. doi:10.1097/00005053-198709000-00004 Reports reliability for experience sampling, in which participants carried a pager and reported their state at signalled moments, with validity evidence from correlations with physiological measures and behavioural indices.
• Stone, A. A. and Shiffman, S. (2002). Capturing momentary, self-report data: A proposal for reporting guidelines. Ann. Behav. Med. doi:10.1207/s15324796abm2403_09 Proposes reporting guidelines because momentary papers were published in ways that made them impossible to assess. It names seven requirements, among them the sampling rationale, the acquisition interface and the compliance rate.
• Stone, A. A., Shiffman, S., Schwartz, J. E., Broderick, J. E. and Hufford, M. R. (2003). Patient compliance with paper and electronic diaries. Controlled Clinical Trials 24(2), 182-199. doi:10.1016/s0197-2456(02)00320-3 Randomised chronic pain patients to paper or electronic diaries recording symptoms three times daily for three weeks, with a hidden photosensor in the paper booklets. Reported compliance of about ninety per cent contrasted with eleven per cent verified on paper and ninety-four per cent electronic.
• Carter, S. and Mankoff, J. (2005). When participants do the capturing. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. doi:10.1145/1054972.1055098 Ran three diary studies in which participants captured experience through photographs, audio, location information and artefacts. The media differed in what they prompted people to record, so the capture technology is part of the method.
• Ho, J. and Intille, S. S. (2005). Using context-aware computing to reduce the perceived burden of interruptions from mobile devices. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. doi:10.1145/1054972.1055100 Built a device that detected activity transitions with wireless accelerometers and delivered messages there or at random times. Messages at transitions were better received, so when a prompt arrives is an experimental variable.
• Czerwinski, M., Horvitz, E. and Wilhite, S. (2004). A diary study of task switching and interruptions. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. doi:10.1145/985692.985715 Ran a week-long diary study with eleven information workers recording tasks and interruptions. Participants reported about fifty task shifts in the week and 0.7 interruptions per task.
• Kahneman, D., Krueger, A. B., Schkade, D. A., Schwarz, N. and Stone, A. A. (2004). A Survey Method for Characterizing Daily Life Experience: The Day Reconstruction Method. Science. doi:10.1126/science.1103572 Introduces the day reconstruction method, in which respondents rebuild yesterday as a sequence of rated episodes. It reports close correspondence with experience sampling for 909 employed women at much lower cost, and is the strongest challenge here to momentary sampling.
• Jones, A., Remmerswaal, D., Verveer, I., Robinson, E., Franken, I. H. A., Wen, C. K. F. and Field, M. (2019). Compliance with ecological momentary assessment protocols in substance users: a meta-analysis. Addiction 114(4), 609-619. doi:10.1111/add.14503 Pooled 126 studies covering 19,431 participants and found mean compliance of about seventy-five per cent, below the recommended rate and unrelated to prompt frequency or assessment length.
• van Berkel, N., Ferreira, D. and Kostakos, V. (2017). The Experience Sampling Method on Mobile Devices. ACM Computing Surveys. doi:10.1145/3123988 Surveys experience sampling as implemented on mobile devices, covering sampling strategies, prompting, compliance and analysis. It finds the configuration reported inconsistently and the sampling scheme's effect largely unexamined.