LLM Sensory Modality Study Proposal
What happens when a language model encounters content meant to feel uplifting, and how can researchers tell whether its internal response reflects more than the words’ familiar meanings? This proposal compares positive sensory inputs with textual descriptions and neutral controls, asking whether modality changes emotion-related representations and downstream behavior. Multimodal models will receive corresponding audio or images, alongside matched textual descriptions; control receive only text. Neutral and mildly unpleasant stimuli will provide comparison conditions, with semantic content, duration, and task context carefully controlled.
A central methodological question concerns the baseline used to identify implicated neural activity. If sampled activations are leveled against ordinary activation, meaningful differences may be obscured or mistaken for general departures from baseline (Zhang & Nanda, 2024). We will therefore preserve and report raw activation distributions, then compare several reference conditions: ordinary task activity, modality-matched neutral stimuli, and within-model responses to the same stimulus across contexts. This will help distinguish stimulus-specific patterns from broad changes in attention, token statistics, or processing load. Emotion probes and targeted activation analyses will be paired with behavioral measures such as calibration, persistence, refusal quality, and susceptibility to reward-hacking rather than treated as evidence of subjective feeling.
The prediction is not simply that “positive” inputs will make a model act more positively. Instead, sensory modality may engage distinguishable representations, and those representations may predict different behavioral effects than equivalent descriptions do. Preregistered contrasts, paraphrase controls, and replication across models will guard against cherry-picking attractive activation maps. By treating baseline choice as an experimental variable rather than a housekeeping step, this study could reveal what current methods smooth away—and what they mistakenly illuminate. It is an inviting possibility: a model’s route to a functional positive-affect pattern may depend not only on what is represented, but on how that representation arrives.
References: