Recognising activity means generalising to a person the model never saw

Year 3 / Module 20 Mobile, ubiquitous and sensor-based computing / Unit 4 / Recognising activity means generalising to a person the model never saw

Recognising activity means generalising to a person the model never saw

HCI3104 · Unit 4 of 10

Activity recognition accuracy is a property of the activity, the sensor placement and the training population together rather than of a classifier, so a headline accuracy figure hides whether the model transferred to a new person or device, and the field's move to deep learning improved within-dataset scores without settling the generalisation problem that deployment actually poses.

Recognising activity means generalising to a person the model never saw

A recogniser's score is not a property of its classifier. It belongs to the activity, the sensor placement and the training population together. Bao and Intille (2004) reported 84 per cent overall accuracy across twenty everyday activities, a number the field has carried ever since, and reported that some activities transferred from other people's data while others needed data from the person being recognised. Transfer, not accuracy, is what a designer needs, and this fourth of ten units asks what a score is worth to the person the model never met.

The score belongs to the whole arrangement

Bao and Intille (2004) collected accelerometer data from twenty people wearing five small biaxial sensors on hip, wrist, upper arm, ankle and thigh, performing everyday tasks with no researcher present to supervise or observe. Decision tree classifiers reached 84 per cent across twenty activities. Four conditions are packed into that figure: the activity set, the five sensor sites, the twenty participants and the annotation protocol. Cutting to two sensors on thigh and wrist dropped performance only slightly.

Anguita and colleagues (2013) released a public dataset. Thirty volunteers performed six activities carrying a waist-mounted smartphone, with a defined train and test partition. A benchmark fixes the population, the placement and the activity set to make comparison possible, and every figure computed on it inherits those conditions. Shoaib and colleagues (2014) had ten participants perform seven activities carrying phones at different body positions, and which sensor led depended on the activity, the position and whether the classifier was personalised.

Anguita and colleagues partitioned windows rather than volunteers, 70 per cent for training and 30 per cent for testing (Anguita et al., 2013). One person's windows can sit on both sides, and the reported 96 per cent overall inherits that, with sitting the weakest class at 88 per cent recall.

A benchmark makes the wrong comparison easy, and the device is the condition it hides

Lara and Labrador (2013) surveyed wearable recognition and found that the vast majority of studies use cross validation on a particular dataset. A score is therefore comparable only against others computed on the same data.

Part of the gap sits in the protocol. Reyes-Ortiz and colleagues (2016) built a transition-aware system that handles the periods between activities and activities unknown to the learning algorithm. They tested it on case studies covering up to thirty-three activities. A continuous stream has no natural boundary, so segmentation decides what counts as a mistake. A label set that omits transitions scores a model on the easy part. An accuracy figure therefore carries two hidden parameters: the segmentation and the label set that produced it. A third hidden condition is the hardware. Stisen and colleagues (2015) tested it with thirty-six phones and watches across thirteen models from four manufacturers, and nine users. Sensor differences impaired performance significantly, and the impairment varied across devices and techniques, so deployed performance was often lower than the published result. Shoaib and colleagues (2016) compared the wrist and the trouser pocket across thirteen activities and seven window sizes. Combining the two positions beat the wrist alone mainly at small windows, where less repetitive activities were hard to separate. Placement, device, window length and activity kind are not independent knobs.

Transfer to a person is the gap the field has only begun to close

Some activities tolerate subject-independent training and others do not, within one study and on one sensor set (Bao and Intille, 2004). A benchmark fixes its population, and an older body is a stranger the model has not met.

Yin and colleagues (2024) systematically reviewed mobile-device recognition and reported the field moving from self-collected data towards shared public datasets for fair comparison. Mitchell and colleagues (2026) released a dataset precisely because data reflecting unconstrained behaviour is scarce, recording twenty healthy participants with inertial measurement units, force-sensing resistor insoles, colour sensors and LiDAR, labelled by activity and terrain.

Hong and colleagues (2024) built CrossHAR, which augments the source sensor data and pretrains a representation in two self-supervised stages, needing no data or labels from the target. It beat the state of the art on unseen target datasets by 10.83 per cent accuracy across several published datasets. The gain is still a benchmark result, but cross-dataset generalisation is now a method line with measured gains rather than only an acknowledged gap.

Deep learning raised the within-dataset score without settling transfer

Hammerla, Halloran and Plotz (2016) ran more than four thousand recognition experiments with randomly sampled configurations across three wearable datasets. Bidirectional long short-term memory networks beat the state of the art on the large benchmark, where more than fifteen per cent mean F1 separated the best and worst architecture, and rankings shifted with the dataset. Deep learning moved the ceiling on a fixed benchmark; it did not make the benchmark represent a stranger. Ordonez and Roggen (2016) show the strongest gain, a convolutional and LSTM stack that fuses sensors and needs no expert features, beating competing deep non-recurrent networks on a public challenge dataset by about four per cent on average. Wang and colleagues (2017) make the trade explicit. Deep models cut the need for hand-crafted features but demand data and computation, and no single model wins everywhere. The open challenges are label transfer from auxiliary domains and lightweight on-device models.

Transfer also has a labelling precondition. Cook, Krishnan and Rashidi (2012) argued that discovery and recognition belong together, because supervised recognition needs predefined labels that a deployment may lack. They showed in the CASAS smart home testbed that finding patterns in classless data improves later recognition.

The contested question: modelling, evidence, or framing?

Three positions are held, and they differ over what to fix. The first holds that transfer is a modelling problem, closed by architectures, regularisation and domain adaptation. Hammerla and colleagues (2016) are the strongest evidence on this side. Ordonez and Roggen (2016) add a fused deep model that needed no hand-engineered features. The remedy is a better model, and success is a higher score on a fixed test set.

The second holds that transfer is a data and evaluation problem, so the score is a property of the protocol. Lara and Labrador (2013) report that within-dataset evaluation is the norm, Anguita and colleagues (2013) show what a benchmark hides, and Reyes-Ortiz and colleagues (2016) show segmentation changing the measured result. Stisen and colleagues (2015) show the device alone moving performance by a large margin, and Yin and colleagues (2024) show that a decade on a score still cannot be compared across studies recognising different activities. The remedy is a held-out population and a held-out device, and success is a score reported with both.

The third holds that transfer is a framing problem, because a deployment does not know its own label set. Cook and colleagues (2012) argue that discovery must precede recognition, since the classes are not given in advance. The remedy is an open label set, and success is a system that finds and names behaviour rather than recognises a fixed list.

What would settle it is a comparison the literature does not yet contain: a conventional baseline and a deep model on one protocol, with a held-out group of people and a held-out set of devices and the activity set fixed in advance. Mitchell and colleagues (2026) move towards this with terrain-labelled data, and Stisen and colleagues (2015) supply the device axis, but neither combines the full comparison.

Rated by kind, the modelling side rests on benchmark experiments (Hammerla et al., 2016; Ordonez and Roggen, 2016) run within one fixed dataset, so the gains are real but bounded. The evaluation side rests on survey evidence (Lara and Labrador, 2013; Yin et al., 2024) and on measured protocol studies (Anguita et al., 2013; Reyes-Ortiz et al., 2016; Stisen et al., 2015). The framing side rests on Cook and colleagues' (2012) argument and on one testbed demonstration. While the dispute is open, a designer reports a recogniser with its held-out population, its devices and its label set. The strongest objection is that the three positions address different products, so no single comparison settles them.

Where this is heading

The direction of travel is transfer by construction rather than by luck. Wang and colleagues (2017) name deep transfer learning and lightweight on-device models as open challenges, tying the accuracy question to the deployment questions that run through the module. Cook and colleagues (2012) supply the longest-running correction, that the classes a deployed system must handle are discovered, not given. A field that reports a score with its population, its devices and its label set can say what its recognisers actually do; one that reports the score alone will keep rediscovering that a model has never met the person it is asked to serve.

Sources

Bao, L. and Intille, S. (2004). Activity Recognition from User-Annotated Acceleration Data. International Conference on Pervasive Computing. doi:10.1007/978-3-540-24646-6_1 Twenty people wore five biaxial accelerometers unsupervised, and decision trees reached 84 per cent over twenty activities; some transferred across people and others did not. Accuracy is a joint property of activity, placement and population.
Lara, O. D. and Labrador, M. (2013). A Survey on Human Activity Recognition using Wearable Sensors. IEEE Communications Surveys and Tutorials. doi:10.1109/surv.2012.110112.00192 Surveyed wearable recognition across sensors, features, methods and evaluation practice. The vast majority of studies use cross validation on a particular dataset, so scores are comparable only against others computed on the same data and the generalisation gap is structural.
Anguita, D., Ghio, A., Oneto, L., Parra, X. and Reyes-Ortiz, J. L. (2013). A Public Domain Dataset for Human Activity Recognition using Smartphones. European Symposium on Artificial Neural Networks. https://www.esann.org/sites/default/files/proceedings/legacy/es2013-84.pdf Published a public benchmark in which thirty volunteers performed six activities carrying a waist-mounted smartphone, with a defined train and test partition. It shows which conditions a benchmark fixes and therefore hides.
Reyes-Ortiz, J. L., Oneto, L., Sama, A., Parra, X. and Anguita, D. (2016). Transition-Aware Human Activity Recognition Using Smartphones. Neurocomputing. doi:10.1016/j.neucom.2015.07.085 Built a transition-aware system handling the periods between activities and activities unknown to the learner, tested on case studies covering up to thirty-three activities. Segmentation changes the measured result, so a score carries hidden protocol parameters.
Stisen, A., Blunck, H., Bhattacharya, S., Prentow, T. S., Kjaergaard, M., Dey, A., Sonne, T. and Jensen, M. (2015). Smart Devices are Different: Assessing and Mitigating Mobile Sensing Heterogeneities for Activity Recognition. ACM Conference on Embedded Networked Sensor Systems. doi:10.1145/2809695.2809718 Tested thirty-six smartphones and smartwatches across thirteen models from four manufacturers with nine users. Sensor differences impaired performance significantly, and deployed performance was often lower than the reported research result.
Shoaib, M., Bosch, S., Incel, O. D., Scholten, H. and Havinga, P. (2014). Fusion of Smartphone Motion Sensors for Physical Activity Recognition. Sensors. doi:10.3390/s140610146 Had ten participants perform seven activities carrying phones at different body positions. Each motion sensor could lead depending on activity, position and whether the classifier was personalised or generalised.
Shoaib, M., Bosch, S., Incel, O. D., Scholten, H. and Havinga, P. (2016). Complex Human Activity Recognition Using Smartphone and Wrist-Worn Motion Sensors. Sensors. doi:10.3390/s16040426 Evaluated motion sensors at the wrist and the trouser pocket across thirteen activities and seven window sizes from two to thirty seconds. Combining the two positions beat the wrist alone mainly at small windows, where less repetitive activities were hard to separate.
Hammerla, N. Y., Halloran, S. and Plotz, T. (2016). Deep, convolutional, and recurrent models for human activity recognition using wearables. International Joint Conference on Artificial Intelligence, 1533-1540. https://ijcai.org/proceedings/2016/220 Ran more than four thousand experiments with randomly sampled configurations across three wearable datasets. Bidirectional long short-term memory networks beat the state of the art on the large benchmark, with more than fifteen per cent mean F1 between best and worst architecture there.
Ordonez, F. J. and Roggen, D. (2016). Deep Convolutional and LSTM Recurrent Neural Networks for Multimodal Wearable Activity Recognition. Sensors. doi:10.3390/s16010115 Combined convolutional and LSTM recurrent layers in a deep architecture that fuses sensors and needs no expert feature design. It outperformed competing deep non-recurrent networks on a public challenge dataset by about four per cent on average and some earlier results by up to nine per cent.
Wang, J. D., Chen, Y. Q., Hao, S. J., Peng, X. H. and Hu, L. (2017). Deep Learning for Sensor-based Activity Recognition: A Survey. Pattern Recognition Letters. doi:10.1016/j.patrec.2018.02.010 Surveyed deep learning for sensor-based recognition across modality, model and application. Deep models cut the need for hand-crafted features but demand data and computation, and the open challenges include transfer learning and lightweight on-device models.
Cook, D., Krishnan, N. C. and Rashidi, P. (2012). Activity Discovery and Activity Recognition: A New Partnership. IEEE Transactions on Cybernetics. doi:10.1109/tsmcb.2012.2216873 Argued that discovery and recognition should be partners, because supervised recognition needs predefined labels a deployment may lack. In the CASAS smart home testbed, discovering patterns in classless data improved later recognition.
Yin, Y. F., Xie, L., Jiang, Z., Xiao, F., Cao, J. N. and Lu, S. L. (2024). A Systematic Review of Human Activity Recognition Based on Mobile Devices: Overview, Progress and Trends. IEEE Communications Surveys and Tutorials. doi:10.1109/comst.2024.3357591 Systematically reviewed mobile-device recognition across activities, sensor data, preprocessing, approaches, evaluation standards and applications. It reports a field moving from self-collected data towards shared public datasets for fair comparison, while noting that scores from studies recognising different activity sets still cannot be compared.
Hong, Z., Li, Z., Zhong, S., Lyu, W., Wang, H., Ding, Y., He, T. and Zhang, D. (2024). CrossHAR: Generalizing Cross-dataset Human Activity Recognition via Hierarchical Self-Supervised Pretraining. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies. doi:10.1145/3659597 Augmented source data and pretrained a representation in two self-supervised stages with no target labels, then beat the state of the art on unseen datasets by 10.83 per cent accuracy. The unseen datasets remain other public label sets.
Mitchell, J. C., Dehghani-Sanij, A. A., Xie, S. and OConnor, R. J. (2026). A Novel Dataset for Gait Activity Recognition in Real-World Environments. Sensors. doi:10.3390/s26030833 Released the Context-Aware Human Activity Recognition dataset, recorded from twenty healthy participants with inertial measurement units, insole force sensors, colour sensors and LiDAR, labelled by activity and terrain. It is recent evidence that the data and transfer problem persists.

Check your understanding & apply it · 4 questions, 1 essay

1A published recogniser reports 96 per cent overall accuracy on a public benchmark whose authors cut the stream into windows and split them randomly into 70 per cent for training and 30 per cent for testing. A team reads the 96 per cent as readiness for a new population. Which option diagnoses the figure?

2A team trains an activity recogniser on a single phone model and predicts that deploying it on thirteen models from four manufacturers will change nothing because the activities are the same. Predicting from this unit, what happens?

3Two engineers disagree about why recognition fails to transfer. One says better architectures will close the gap; the other says transfer is a data and evaluation problem. Adjudicating from this unit, which option is correct?

4A team evaluates its recogniser on steady-state segments only, omitting the transitions between activities, and reports the resulting accuracy as the product's figure. Judging from this unit, what is wrong and what should change?

5Essay A team has built a phone-based activity recogniser and reports 94 per cent accuracy on a public benchmark dataset. They plan to ship it to a general user base on whatever phones those users own. Using this unit's sources, advise the team on what their figure does and does not establish, what evidence would show the recogniser generalises, and how they should report their result. Cite the sources you rely on at each step.

The first thing to say is that the figure establishes performance under the conditions the benchmark fixed, and nothing more. Anguita and colleagues (2013) show what a benchmark fixes: thirty volunteers, six activities and a waist-mounted phone. Every score computed on that partition inherits those choices, so 94 per cent is a statement about that population, that placement and that activity set. Bao and Intille (2004) show how much those choices matter, since some activities were recognised well from other people's data while others required the wearer's own data, and cutting to two sensors cost little. The team should therefore expect the number to move as soon as the population or the placement changes. The device axis is the second problem. Stisen and colleagues (2015) tested thirty-six smartphones and smartwatches across thirteen models from four manufacturers and found that sensor and sensor-handling differences impaired performance significantly, with deployed performance often lower than the reported research result. A recogniser shipped to whatever phones users own will meet exactly that heterogeneity, so the benchmark score is not a deployment estimate. The team should also check the protocol behind the number. Reyes-Ortiz and colleagues (2016) show that segmentation and the treatment of transitions change the measured result, and Lara and Labrador (2013) report that within-dataset evaluation is the norm and that this practice overstates performance. The evidence that would show generalisation is a held-out evaluation: train on one group of people and one set of devices, test on people and devices the model never saw, and report the result per participant and per device rather than as one pooled figure. Mitchell and colleagues (2026) move in this direction by releasing real-world data labelled by activity and terrain, and Yin and colleagues (2024) confirm that most studies still evaluate within self-collected datasets, so this test is still rare. If the team adds a deep model to raise the score, Hammerla and colleagues (2016) and Ordonez and Roggen (2016) show that the gain is real but measured within the training conditions, and Wang, Chen and colleagues (2017) note that no single model wins everywhere and that transfer remains an open challenge. The recommendation is to report the score with its population, its devices, its activity set and its segmentation, and to treat the benchmark figure as a ceiling under chosen conditions rather than as a promise about strangers.