Year 4 / Module 27 Designing and evaluating adaptive and intelligent systems / Unit 3 / The technique family fixes the justification the interface can give

The technique family fixes the justification the interface can give

HCI4103 · Unit 3 of 10

A recommender can only justify itself in terms of the mechanism it was built on, and because collaborative filtering can explain an item by resemblance to other people while content-based and knowledge-based methods can explain it by its own attributes, the explanatory reach of the interface is settled by an algorithmic choice that teams usually make on accuracy grounds alone.

The technique family fixes the justification the interface can give

Every recommendation is an answer to a question the person did not quite ask. The argument of this unit is that the mechanism, chosen earlier, settles which reason is available. A collaborative filter can say that people like you chose this item. A content-based recommender can say that the item carries attributes you have favoured. A matrix factorisation model can say neither until somebody builds a second system to name its factors. Teams make the choice on measured accuracy, and they rarely price the explanatory consequences when they make it.

The claim is about the default, not about possibility: the reason an interface gets for free matches its mechanism, and any other reason is additional construction. Nor is it a claim that explanations fail, since their effects depend on the person and on presentation. The third of ten units, this one asks what the chosen technique fixes about the reason the interface can give. The fourth unit asks what the interface should show when it knows nothing about the person yet.

The founding problem is whose judgement a recommendation carries

Resnick and Varian (1997) introduce recommender systems as systems that gather other people's evaluations and aggregate them for the person seeking advice. They set out the incentive problems that follow. A person can free-ride on a profile others built, and content owners can praise their own material and disparage a competitor's. They add a privacy concern, since a profile of interests is itself sensitive. Justification is therefore the founding design problem: the aggregation that makes a recommendation useful also hides its basis. The person receiving it cannot see whose judgement it carries.

Resemblance is the reason a collaborative filter can give

Sarwar and colleagues (2001) built the item-based variant of collaborative filtering, computing similarity between items rather than between users. They reported that their item-based algorithms outperformed the user-based nearest-neighbour approach and addressed the scale and sparsity problems of a growing site. A recommendation is therefore a resemblance between the item offered and one the person already touched, which is what the interface can report.

Linden, Smith and York (2003) record the deployed form at Amazon. They compared item-to-item collaborative filtering with cluster models and search-based methods. Their algorithm produces recommendations in real time and scales independently of the number of customers and items, and the interface that followed is the familiar statement that customers who bought this item also bought that one. The label names other shoppers, not the item's properties, because resemblance is the only reason the mechanism has.

Koren, Bell and Volinsky (2009) present matrix factorisation as superior to classic nearest-neighbour techniques and able to absorb implicit feedback, temporal effects and confidence levels, in work driven by the Netflix Prize. A factorisation model represents a person and an item as latent vectors whose numbers predict ratings well and carry no name. An interface built on it has nothing to print until somebody supplies a labelling step.

Attributes are the reason a content-based method can give

Pazzani and Billsus (2007) describe the family that can name the item. In content-based recommendation the method represents each item by its attributes and matches it to a profile of the person's interests, which the method maintains as evidence arrives. The interface can then offer a reason of the form that this item resembles what you chose before. The chapter states the limits: the reason is only as good as the item metadata, and the method tends to recommend more of what the person has already seen.

Burke (2002) surveys hybrid systems and the ways a designer combines content-based and collaborative signals. The choice of combination decides which reason the system can offer, so a hybrid weighting collaborative evidence heavily inherits the resemblance justification.

The explanation literature moved the question from mechanism to presentation

Herlocker, Konstan and Riedl (2000) opened a different question, describing collaborative filtering as a black box and building explanation interfaces that expose the reasoning and data behind a recommendation. Their experiments reported which components were most compelling and that explanations could improve acceptance.

Their own study compared explanation interfaces, including a histogram of how the nearest neighbours had rated the item, and varied how much of the algorithm the interface exposed. The simplest explanations were the most compelling, while a detailed account of the algorithm scored lower, which puts the effective variable in the presentation rather than the mechanism. The limit is the measure: the study reported acceptance and preference, not whether the person could check the reason against the item.

Tintarev and Masthoff (2007) set out seven possible advantages an explanation facility can deliver, and examined how presentation shapes them. Their handbook chapter argues that an evaluation must state which aim it measures, because the aims do not move together: transparency and persuasion are different targets, and optimising one may lose the other. Presentation is also where a written reason meets a reader in a second language. Their own studies then tested that demand: personalising a feature-based explanation was, contrary to expectation, detrimental to effectiveness while tending to improve satisfaction (Tintarev and Masthoff, 2012).

The contested question: does the mechanism settle what the interface can say?

The first position holds that the mechanism determines most of the interface. The deployed language of collaborative filtering is resemblance, because the mechanism compares items through the people who touched them (Linden et al., 2003), and the leading accuracy method moved to latent factors that carry no name (Koren et al., 2009). Content-based methods can name attributes, because they represent items by them (Pazzani and Billsus, 2007). In a hybrid the dominant signal decides the available reason (Burke, 2002). The reason that reaches the person is further shaped above the engine, since what deployed recommenders recommend concentrates on a small set of popular items (Jannach et al., 2015). On this reading the interface is downstream of choices usually made on accuracy.

The second holds that explanation is a layer with degrees of freedom of its own, settled by presentation and person rather than mechanism. Herlocker and colleagues (2000) found that components differed in how compelling they were and that explanations changed acceptance, which puts the effective variable in the presentation. Millecamp and colleagues (2019) found that the benefit of explanations for music recommendations depended on need for cognition. McAuley and Leskovec (2013) show the repair is available in principle, since combining ratings with review text gave interpretable labels for latent rating dimensions.

What would settle it is not in this literature. The decisive pair would hold the algorithm fixed and vary only the explanation's basis, then hold the explanation fixed and vary the mechanism, measuring in each case whether the person can check the reason against the item. The studies here report acceptance, preference and confidence, none of it verification. Until that measurement exists, the defensible statement is narrower: the technique sets the default reason, and the default can be bought back with work.

The mechanism position rests on documented deployed systems (Linden, Smith and York, 2003) and a benchmark comparison of algorithm families (Sarwar et al., 2001); the layer position rests on controlled experiments that measured acceptance, preference and confidence (Herlocker et al., 2000; Millecamp et al., 2019). While the dispute is open, choose the mechanism for the reason it can print, and test whether a person can check that reason. The strongest objection is that no study here pairs an explanation's basis with the mechanism at the same scale.

Where this is heading

The direction of travel is that the reason becomes the output of a separate model rather than a report of the mechanism. Konstan and Riedl (2012) moved the question from algorithms to user experience, and Tintarev and Masthoff (2010) demanded that a designer evaluate an explanation against a named aim. McAuley and Leskovec (2013) showed that a researcher can give a latent dimension a name from text. Against that, Jannach and colleagues (2015) remain the corrective: the reason on display describes a selection made for reasons the interface may not state. The separate model is now a language model, and its arrival is thinner than its fluency suggests. Said (2025) searched the literature from the launch of ChatGPT to November 2024 and found only six studies that directly address explanations generated by a large language model. The reviewed work reports that people prefer the generated reasons, which is a preference rather than a check. The open question is unchanged: whether the person receiving the reason can check it, since a fluent reason and a grounded one look the same on the screen.

Sources

  • Resnick, P. and Varian, H. R. (1997). Recommender systems. Communications of the ACM. doi:10.1145/245108.245121 Introduced recommender systems as systems that aggregate other people's evaluations, and named free-riding, self-serving contributions by content owners and privacy as the problems that follow. It makes justification the founding design problem.
  • Sarwar, B., Karypis, G., Konstan, J. and Riedl, J. (2001). Item-based collaborative filtering recommendation algorithms. Proceedings of the 10th international conference on World Wide Web. doi:10.1145/371920.372071 Introduced item-based collaborative filtering, computing similarity between items to meet the scale and sparsity problems of a growing site, and reported that the item-based algorithms outperformed the user-based nearest-neighbour approach. The new axis makes the recommendation an item-to-item resemblance, which is the reason the interface can report.
  • Koren, Y., Bell, R. and Volinsky, C. (2009). Matrix Factorization Techniques for Recommender Systems. Computer. doi:10.1109/MC.2009.263 Presented matrix factorisation as superior to classic nearest-neighbour techniques and able to absorb implicit feedback, temporal effects and confidence levels, in work driven by the Netflix Prize. Its factors carry no name, so the interface has nothing to print without a labelling step.
  • Pazzani, M. J. and Billsus, D. (2007). Content-based recommendation systems. In The Adaptive Web: Adaptation Methods and Strategies, Lecture Notes in Computer Science. doi:10.1007/978-3-540-72079-9_10 Described content-based recommendation, in which each item is represented by its attributes and matched to a profile of the person's interests. It names the family's limits, the quality of the item metadata and a tendency to recommend more of what has been seen.
  • Burke, R. (2002). Hybrid Recommender Systems: Survey and Experiments. User Modelling and User-Adapted Interaction. doi:10.1023/A:1021240730564 Surveyed hybrid systems, classifying how content-based and collaborative signals are combined, with experiments for each class. The combination decides which signal dominates, and the dominant signal decides which reason the system can offer.
  • Herlocker, J. L., Konstan, J. A. and Riedl, J. (2000). Explaining collaborative filtering recommendations. Proceedings of the 2000 ACM conference on Computer supported cooperative work. doi:10.1145/358916.358995 Described collaborative filtering as a black box with no transparency into how a recommendation was produced, and built explanation interfaces that expose the reasoning and data behind it. The experiments reported which components were most compelling and showed that explanations could improve acceptance.
  • Tintarev, N. and Masthoff, J. (2007). A Survey of Explanations in Recommender Systems. 2007 IEEE 23rd International Conference on Data Engineering Workshop. doi:10.1109/ICDEW.2007.4401070 Reviewed explanation in recommender systems, setting out seven possible advantages an explanation facility can deliver and how existing measures can evaluate explanation quality. It examines how presentation shapes explanations, making the explanation a design object distinct from the mechanism.
  • Tintarev, N. and Masthoff, J. (2010). Designing and Evaluating Explanations for Recommender Systems. Recommender Systems Handbook. doi:10.1007/978-0-387-85820-3_15 Reviews how explanations have been designed and evaluated, and argues that an evaluation must state which aim it measures because the aims do not move together. It supplies the methodological demand against which later explanation studies are read.
  • Tintarev, N. and Masthoff, J. (2012). Evaluating the effectiveness of explanations for recommender systems: generic framework and empirical study. User Modelling and User-Adapted Interaction. doi:10.1007/s11257-011-9117-5 Four studies in two domains measured effectiveness and satisfaction together. Personalisation was, contrary to expectation, detrimental to effectiveness while tending to improve satisfaction, so the aims came apart in practice. It is the primary study behind the unit's claim that the aims conflict.
  • Linden, G., Smith, B. and York, J. (2003). Amazon.com recommendations: item-to-item collaborative filtering. IEEE Internet Computing. doi:10.1109/MIC.2003.1167344 Compared item-to-item collaborative filtering with traditional collaborative filtering, cluster models and search-based methods, and reported that their algorithm recommends in real time and scales independently of the number of customers and items. It records the interface convention that names other shoppers rather than item attributes.
  • Konstan, J. A. and Riedl, J. (2012). Recommender systems: from algorithms to user experience. User Modelling and User-Adapted Interaction. doi:10.1007/s11257-011-9112-x Argued that recommender research should move from algorithmic accuracy to user experience, and reviewed how interface and interaction decisions condition whether a recommendation is worth anything. It sets the agenda that later explanation work works within.
  • Jannach, D., Lerche, L., Kamehkhosh, I. and Jugovac, M. (2015). What recommenders recommend: an analysis of recommendation biases and possible countermeasures. User Modelling and User-Adapted Interaction. doi:10.1007/s11257-015-9165-3 Analysed what several recommenders recommend across datasets and reported strong popularity bias, with a small set of items carrying most recommendations, then tested re-ranking countermeasures. The reason on display describes a selection made on top of the mechanism.
  • McAuley, J. and Leskovec, J. (2013). Hidden factors and hidden topics: understanding rating dimensions with review text. Proceedings of the 7th ACM conference on Recommender systems. doi:10.1145/2507157.2507163 Combined latent rating dimensions with latent topics from review text, and reported interpretable textual labels for the latent dimensions and more accurate prediction, especially for new products and users. It is the route from an uninterpretable factor to a nameable aspect.
  • Millecamp, M., Htun, N. N., Conati, C. and Verbert, K. (2019). To explain or not to explain: the effects of personal characteristics when explaining music recommendations. Proceedings of the 24th International Conference on Intelligent User Interfaces. doi:10.1145/3301275.3302313 Tested explanations in a music recommender with a within-subject online study measuring personal characteristics alongside qualitative and quantitative evaluation. Explanations helped participants with a low need for cognition most, and for those with a high need for cognition could create a lack of confidence, so the benefit is conditional on the person.
  • Said, A. (2025). On explaining recommendations with Large Language Models: a review. Frontiers in Big Data. doi:10.3389/fdata.2024.1505284 Searched the literature from the launch of ChatGPT to November 2024 and found only six studies that directly address explanations generated by a large language model. The reviewed work reports that people prefer the generated reasons, so the area is early and preference is not verification.

Check your understanding & apply it · 4 questions, 1 essay

1A product note says: "Our engine is a latent-factor model, so the interface can name the aspects of an item that a person liked." Which diagnosis does the unit support?
2A team weights the collaborative signal heavily in a hybrid because it scores better offline. What does the unit predict the interface will be able to say by default?
3Two reviewers disagree about an explanation study. One says the measured effect proves the mechanism does not constrain the interface. The other says the mechanism sets the reason and presentation changes only its force. Which verdict does the unit support?
4A team proposes to settle whether its printed reason matches the engine by asking users how convincing they find it. Which judgement does the unit support?
5Essay A team is shipping a recommender whose engine is a matrix factorisation model. The product manager wants the interface to tell each user why an item was recommended, and proposes printing a short sentence in the pattern this was recommended because you liked the author, generated by a language model from the item's description. An engineer objects that the sentence will not correspond to anything the model actually used, so it is a fiction dressed as a reason. Using this unit's sources, assess both positions. State what the engine family does and does not make available as a reason, say what the explanation literature establishes about the effects of an explanation and why that bears on the engineer's objection, and set out the study the team would need to run to decide whether the printed reason should stay.