# Arctop — Full Content

> Full text of every deep-dives post on arctop.com. See https://arctop.com/llms.txt for the curated index.

---

# Reinforcement Learning from Brain Feedback (RLbF) for Large Language Model Improvement
URL: https://arctop.com/deep-dives/rlbf-reinforcement-learning-from-brain-feedback
Published: 2026-05-05

This paper is available on Zenodo <a href="https://doi.org/10.5281/zenodo.20043908">here</a>.

<p><strong>Authors:</strong> Dan Furman*, Eitan Kay, Ben Kogan, Kuan-Jung Chiang. <strong>Affiliation:</strong> Arctop Inc.<br />*Corresponding Author, Email: df@arctop.com</p>

<h2>Abstract</h2>

Large Language Models (LLMs) aligned using Reinforcement Learning from Human Feedback (RLHF) are learning <em>what</em> to say from discrete, voluntary preference judgments, but not <em>how</em> their communication lands. The LLM does not know how different answers affect a listener's cognition and emotion in real time, regardless of how intelligent the model is on benchmark tests. This becomes a major gap in developing trust programmatically. This communication gap can be closed by introducing a promising signal for enriching temporal information in the RLHF training process: the brain electroencephalography (EEG) measurement. EEG's high temporal resolution makes it especially well-suited for the purpose of contextualizing time series data, with multiple data points per second making within-response changes in listener states potentially observable.

Most EEG foundation models today have been developed as general EEG representation learners for downstream decoding tasks rather than as alignment systems for LLM models. The field still lacks large, ecologically valid datasets that couple natural conversation with time-resolved cognitive-state labels. Against that background, we propose Reinforcement Learning from Brain Feedback (RLbF), an LLM post-training framework that uses calibrated cognitive-state predictors to convert decoded EEG signals into a continuous, involuntary, noisy reward source for language-model adaptation. RLbF formalizes communication as a Partially Observable Markov Decision Process and defines a three-component reward function combining prediction accuracy, cognitive resonance, and application-layer objectives. A three-phase training pipeline progresses from supervised fine-tuning through prediction model calibration to reinforcement learning with multidimensional empathic reward.

We hypothesize that responsible RLbF deployment could also act as a data engine, with real-world conversational use generating aligned neuro-conversational traces that later support improved cognitive-state decoders and future EEG foundation models optimized for interactive environments. We hypothesize that models trained with brain-based reward signals may acquire communication skills that persist even when EEG is no longer available at inference time. If so, brain feedback could serve as a training signal for more empathic and effective language models without requiring end users to wear EEG hardware during deployment at scale. We instantiate this framework in a proof-of-concept platform, Isaac, which implements the proposed closed-loop cognitive feedback architecture for experimental study. We also outline an initial evaluation protocol designed to support pre-registered testing and to examine key ethical questions, including the boundary between empathic and persuasive computing.

<blockquote>
<p><em>"A robot may not injure a human being or, through inaction, allow a human being to come to harm." —</em> Isaac Asimov.<br />
<em>Law #1. Handbook of Robotics, 56th Edition, 2058 A.D.</em></p>
</blockquote>

<h2>1. Introduction</h2>

Large Language Models (LLMs) are now embedded in digital applications that interact with people every day, and related model classes are increasingly moving off the screen and into the ambient physical world through embodied AI and robotics. At the same time, many deployed systems remain poorly aligned with user well-being, often relying on engagement dynamics reminiscent of the "dopamine fracking" that shaped early social media platforms. As AI becomes more deeply woven into both software and the built environment, new frameworks, capabilities, and data architectures are needed to improve model behavior and enable the safe, scalable deployment of powerful AI systems.

<Figure src="/deep-dives/pics/deep-dives/rlbf_figure1.webp" alt="Comparison of Alignment Paradigms: Standard RLHF vs. Proposed RLbF." caption={"<strong>Figure 1. Comparison of Alignment Paradigms: Standard RLHF vs. Proposed RLbF.</strong>"} />

<h3>1.1 The Problem: Language Models Cannot Sense Their Impact</h3>

Modern large language models are remarkably adept at generating fluent, informative, and contextually appropriate text. Through Reinforcement Learning from Human Feedback (RLHF), LLM outputs have been further shaped to produce responses that people tend to prefer, often increasing engagement, time spent, and repeated daily use (Ouyang et al., 2022), potentially driving dependence. Yet these models operate with a fundamental limitation: they cannot directly perceive how their words affect the person reading or listening. They have no biological grounding in human cognition and therefore no intrinsic access to the lived process of hearing language, interpreting it, and responding to it as a human does. In other words, LLMs are computational systems that can approximate human expression without sharing the underlying neural processes that produce and receive language in the brain. Similar outputs do not imply similar mechanisms, and this gap may prove to be an important frontier in the divergence between machine and human intelligence.

When a skilled human communicator explains a complex idea, they continuously monitor their listener, watching for cues such as furrowed brows that signal confusion, glazed eyes that suggest overload, nods that confirm understanding, or tension that indicates distress. These signals are often involuntary, continuous, and pre-reflective, unfolding naturally from moment to moment within a feedback loop that allows the communicator to adapt in real time: simplifying when the listener is struggling, expanding when they are engaged, and offering warmth when they are stressed. This ability to sense and respond to a communication partner's cognitive and emotional state is the foundation of empathic communication. LLMs have no access to this bioinformational channel.

When an RLHF-trained model generates a response, it receives no direct signal about whether that response overwhelmed the user, struck the wrong emotional tone, or caused the listener to disengage partway through. The feedback available to the model is instead post hoc: voluntary, discrete, sparse, and delayed (Section 1.2 details these structural limitations). The rich, continuous, and largely involuntary signals that make human communication adaptive are therefore absent from the training loop. As a result, potentially informative traces of a user's internal cognitive and affective state are lost, even though they may unfold in real time during the interaction. This information gap is a fundamental limitation that is unlikely to be solved simply by scaling models, improving prompting, or replacing human feedback with synthetic feedback. It is a structural property of the RLHF paradigm itself, and one that directly shapes the capabilities and limits of the models trained under it.

<h3>1.2 Relevant Background and the Missing Interface</h3>

Two research literatures are especially relevant to this problem. First is RLHF, which provides a successful post-training framework for adapting LLM behavior, and second is EEG foundation models which aim to learn reusable EEG representations for downstream decoding tasks. RLbF sits at the interface between these literatures: it requires a practical way to infer cognitive state from EEG, and a training framework capable of using those estimates to shape language-model behavior, generalize across individuals, and support real-time adaptation and personalization.

<strong>EEG foundation models.</strong> The foundation model paradigm — pretrain on large unlabeled data, then fine-tune for specific tasks — transformed natural language processing (NLP) and computer vision, and it has inspired a parallel effort in EEG. This literature matters for RLbF because any practical brain-feedback system needs a decoder that can operate across users, sessions, and recording conditions. EEG foundation models pursue that robustness by learning general EEG representations that can later be adapted to downstream tasks such as abnormal detection, event classification, sleep staging, emotion recognition, and workload estimation. Our systematic review (Section 2.4), however, identifies two limitations especially relevant here: pretrained EEG representations often transfer weakly under frozen evaluation, and the field still lacks large-scale ecologically valid datasets that couple natural conversation with time-resolved cognitive-state labels under deployment-like conditions.

Across this literature, architecture and objective design often appear to matter more than scale alone in predicting downstream performance, and classical methods remain competitive on several clinical tasks (Lotte et al., 2018). For the purposes of this paper, the key implication is that decoder quality remains an open bottleneck, while interaction-coupled cognitive-state data remain scarce.

<strong>RLHF limitations.</strong> RLHF has been successfully deployed in large-scale production assistants and has improved the helpfulness, instruction-following, and safety of modern language models. Yet the framework retains three structural limitations. First, feedback is discrete and sparse: a complex communicative exchange is often reduced to a single preference judgment, collapsing rich temporal dynamics into a coarse supervisory signal. Second, feedback is voluntary, reflective, and interface-mediated: it is filtered through conscious deliberation, comparison framing, and the mechanics of selecting which response is better, introducing annotation artifacts, inter-annotator disagreement, and social desirability bias. Third, feedback is coarse in temporal granularity: it typically cannot identify which sentence or moment within a response produced a positive or negative reaction. These limitations are intrinsic to the paradigm rather than mere artifacts of scale or implementation.

A related concern is that because human preference judgments often favor flattering or agreeable responses over more truthful ones, preference-based optimization entrenches AI sycophancy despite potential risks. Sycophantic responses can shape LLM user attitudes towards increased righteousness and reduced willingness to repair relationships (Cheng et al., 2026). While the effect of AI sycophancy on the general population's judgments and behaviors remain an under-explored topic, it is clear that the development of effective safety guardrails is needed alongside national security countermeasures against threats from cybersocial engineering and deepfake technology.

<h3>1.3 RLbF as Cognitive-State Feedback for Post-Training</h3>

RLbF uses EEG-derived cognitive-state estimates as a feedback layer for language-model post-training. In this architecture, EEG decoding and language-model alignment are separate components: a decoder maps raw EEG into estimated cognitive states, and the language model is optimized against those estimates. This separation lets the language model benefit from improvements in EEG decoding without requiring the language model itself to process raw EEG.

This design is practical in the near term because it relies on calibrated cognitive-state predictors built with conventional supervised pipelines, while remaining fully compatible with future EEG foundation models. More importantly, this architecture establishes a self-reinforcing data engine. Responsible RLbF deployments will generate precisely what the neurotechnology field currently lacks: large-scale, ecologically valid datasets that couple continuous neural traces with conversation context, model outputs, and outcomes. By synthesizing RLHF's post-training machinery with continuous, involuntary neurophysiological feedback, RLbF treats cognitive-state estimates as a noisy-but-informative reward signal that complements discrete preference judgments. Ultimately, interacting with the agent generates the aligned neuro-conversational logs needed to bootstrap the next generation of EEG foundation models, creating a flywheel where current deployments yield the data necessary for future decoder improvements.

<Figure src="/deep-dives/pics/deep-dives/rlbf_figure2.webp" alt="The RLbF Data Engine and Ecologically Valid Dataset Generation." caption={"<strong>Figure 2. The RLbF Data Engine and Ecologically Valid Dataset Generation.</strong>"} />

<h3>1.4 Contributions</h3>

This paper makes a dual contribution to this emerging area of AI performance research: <strong>RLbF</strong>, a general theoretical framework for neuro-alignment of language models, and <strong>Isaac</strong>, a proof-of-concept platform that instantiates the framework to collect ecologically valid conversational-neurophysiological data. Together they support the following specific contributions:

<ol>
<li><strong>The RLbF paradigm.</strong> We formalize Reinforcement Learning from Brain Feedback as a method for LLM post-training, defining the cognitive state space, the POMDP interaction loop, the temporal delay model, and a three-component reward function (prediction accuracy, cognitive resonance, application-layer objectives) with mathematical specifications and testable hypothesis for experimentalists.</li>
<li><strong>The feedback-layer framing.</strong> We articulate an architecture in which decoded cognitive-state estimates serve as reward signals for LLM post-training, while leaving the decoder implementation open to classical models, proprietary systems, or future EEG-foundation-model-based backbones.</li>
<li><strong>The first cognitive-state-conditioned LLM pipeline.</strong> We specify a complete three-phase training pipeline – supervised fine-tuning on synthetic cognitive-state-conditioned data, calibration of cognitive-state predictors on real EEG session recordings, and reinforcement learning with the full empathic reward – with success criteria, failure modes, and data requirements.</li>
<li><strong>A cognitive state transition prediction model.</strong> We define a learned predictor that maps (conversation context, current cognitive state, utterance) to predicted next cognitive state, serving both as the prediction accuracy reward component and the mechanism for open-loop inference from novel, high temporal resolution, multidimensional data.</li>
<li><strong>The open-loop empathic transfer hypothesis.</strong> We propose and theoretically ground the hypothesis that LLM communication skills learned from brain feedback during training will persist at inference, even when EEG hardware is removed. We design a controlled experiment to test this hypothesis with suitable statistical analyses that, if successful, would enable deployment to users without neurological instrumentation or head-based wearables for equally empathetic and effective interactions.</li>
<li><strong>An empathic computing ethical framework.</strong> We provide an ethical framework that distinguishes empathic computing from persuasive computing through testable criteria, addresses informational power asymmetry, neurorights, mental privacy, and cognitive dependence. We also provide a 14-point review checklist for RLbF deployments.</li>
<li><strong>The Isaac platform.</strong> We describe the proof-of-concept implementation (Isaac) that realizes the closed-loop cognitive feedback architecture, including its dual-agent design as a data collection platform for aligned conversational-neurophysiological sessions and its proposed evolution toward a single post-trained empathic agent.</li>
</ol>

<h3>1.5 Paper Organization</h3>

This paper proceeds from motivation to mechanism to validation. Section 2 situates RLbF at the intersection of RLHF, affective computing, EEG-LLM systems, and EEG foundation models, clarifying why continuous cognitive-state feedback is appealing while decoder quality and interaction-coupled data remain bottlenecks. Sections 3 through 5 then move from theory to implementation: Section 3 formalizes the RLbF framework and reward structure, Section 4 presents the Isaac platform as the concrete implementation of that framework, and Section 5 lays out the three-phase training pipeline. Sections 6 and 7 turn from construction to testing. Section 6 defines the open-loop empathic transfer hypothesis, which is this paper's central empirical claim, and Section 7 specifies the evaluation protocol designed to falsify or support it. Section 8 then addresses the ethical boundary conditions for any RLbF deployment, Section 9 discusses the broader implications and limitations of the framework, and Section 10 closes by summarizing RLbF as both an alignment proposal and a hypothesis-generating research.

<h2>2. Background and Related Work</h2>

RLbF draws primarily on five domains of prior work, summarized below, while identifying the limitations that RLbF addresses and distinguishing the proposed approach from existing methods that have led to the state-of-the-art LLMs today.

<h3>2.1 Reinforcement Learning from Human and AI Feedback</h3>

The modern paradigm of aligning language models with human preferences emerged from Christiano et al. (2017), who demonstrated that RL agents could learn complex behaviors from pairwise preference comparisons. The framework was extended to language by Ziegler et al. (2019), scaled by Stiennon et al. (2020), and consolidated in InstructGPT (Ouyang et al., 2022). Constitutional AI (Bai et al., 2022) and RLAIF (Lee et al., 2023) replaced human annotators with AI feedback. Direct Preference Optimization (Rafailov et al., 2023) simplified the pipeline further.

Despite their success, these approaches share the structural limitations described in Section 1.2: discrete, sparse, voluntary feedback at conversation-level granularity. RLbF replaces this with continuous, multidimensional neurophysiological signals at sub-second temporal resolution. At the same time, RLHF has several important advantages: it requires no specialized hardware, can aggregate preferences from thousands of diverse annotators, has an established empirical track record at scale, and directly trains for content quality, including helpfulness, truthfulness, and harmlessness. By contrast, RLbF is too new to claim validation at this scale; the capabilities proposed here remain theoretical and require empirical testing. RLbF is therefore best understood as complementary to, not a replacement for, content-quality training. In this framing, RLHF helps models learn what to say, while RLbF may help them learn how to say it.

<h3>2.2 Affective Computing and Neurofeedback</h3>

Affective computing (Picard, 1997; Calvo and D'Mello, 2010) established the broader premise that computational systems can detect and respond to human affective state. In educational settings, this line of work informed affect-aware tutoring systems, where adaptation to learner emotion and engagement has been associated with improved learning outcomes (D'Mello and Graesser, 2012). However, prior work uses rule-based or heuristic adaptation mechanisms – hard coded mappings from detected states to predefined strategies. RLbF differs fundamentally: the model learns its adaptation strategy through optimization, discovering effective communication patterns rather than having them programmed.

Closed-loop systems using neurodata (Sitaram et al., 2017; Muhl et al., 2014) have demonstrated that continuous involuntary neural signals can drive real-time systems, but they adapt stimulus parameters (task difficulty, reward thresholds), not generative language. A conceptual precursor to RLbF's use of neural signals as RL reward can be found in BCI adaptation driven by error-related potentials (ErrP) (Fidencio et al., 2025), wherein ErrPs serve as reward signals for reinforcement learning in motor BCI control. This work demonstrates the viability of brain computer interface (BCI)-mediated content adaptation, including the Arctop empathic computing platform that uses real-time brain-state (cognitive state, in our terminology) decoding to modulate content delivery (Furman and Kwalwasser, US20210390366A1, 2021). RLbF extends this from content modulation to content generation: teaching a language model to produce different text as a function of the listener's cognitive state through post-training.

<h3>2.3 Recent EEG-LLM Integration Efforts</h3>

The EEG–LLM interface literature is still emerging. Zhang et al. (2026), for example, introduce a brain-LLM interface that uses EEG-derived satisfaction estimates to guide image generation at test time. This is among the closest existing work to RLbF, but the proposed framework differs in a fundamental respect: Zhang et al. use EEG as an inference-time control signal, whereas RLbF proposes using EEG during training as a continuous reward source for shaping language generation. Relatedly, their approach is organized around user-satisfaction estimation, while RLbF proposes a richer, multidimensional cognitive-state feedback mechanism for reward shaping over time.

ARIEL (Sorino et al., 2024) was a working BCI+LLM system that used real-time EEG-based emotion recognition to guide an LLM conversational agent for emotional support. When the emotion recognizer detected a negative emotional state from EEG signals, ARIEL initiated a supportive dialogue, with the LLM's behavior steered via role-play prompting that incorporated the recognized emotion label. ARIEL demonstrated that EEG-driven LLM interaction is feasible, but it differed from RLbF on three axes: (a) it operates at test time via prompt injection rather than at training time via reward shaping – the LLM weights are unchanged by the EEG signal; (b) it classifies EEG into discrete emotion categories rather than decoding continuous multi-dimensional cognitive state; and (c) it uses rule-based prompt formatting to steer the LLM rather than learned RL adaptation. RLbF's contribution is the shift from test-time prompting to training-time optimization, enabling the model to internalize empathic communication patterns rather than relying on external prompt engineering.

A related line of work uses neural data to inform LLM fine-tuning: brain-informed alignment approaches (Bilgin et al., 2026) leverage fMRI and EEG features to guide supervised fine-tuning of language models. These methods differ from RLbF in that they use offline neural data for supervised optimization rather than real-time neural signals as a continuous RL reward during post-training. Several surveys (Chandrasekharan and Jacob, 2025; Babu et al., 2025) and additional work on EEG-to-text generation (Mishra et al., 2024) and predictive communication (Caria, 2025) map the broader landscape. RLbF's specific contribution is using multi-dimensional cognitive state as a novel, continuous reward signal for post-training.

<h3>2.4 EEG Foundation Models: A Systematic Review</h3>

The foundation model paradigm – pretraining a large model on unlabeled data, then fine-tuning for specific downstream tasks – transformed NLP and computer vision. EEG is, in principle, well suited to this paradigm: clinical archives contain tens of thousands of hours of unlabeled recordings, labeled EEG data is scarce and expensive, and the diversity of EEG applications (seizure detection, sleep staging, BCI control, emotion recognition, cognitive assessment) creates demand for a general-purpose representation. Yet EEG poses challenges with no analogue in text or images: high inter-subject variability, low signal-to-noise ratios, and extreme heterogeneity in recording setups (channel counts ranging from 2 to 256, varying electrode positions, different sampling rates).

We reviewed 12 papers published between January 2021 and April 2026, identified through systematic search of arXiv, PubMed, IEEE Xplore, and Semantic Scholar, and supplemented by citation chaining from benchmark papers. Inclusion required that a paper perform self-supervised pretraining on EEG data and demonstrate transfer to at least one downstream task. The review employed an AI-assisted iterative process over three research cycles (initial taxonomy construction, cross-reference analysis with gap identification, and validation with refinement), with human oversight at cycle boundaries to verify key findings; a limitation of this methodology is reliance on available text rather than experimental reproduction. The 12 papers comprise: eight core EEG foundation models (BENDr, BIOT, LaBraM, EEGPT, CBraMod, EEGMamba, NeuroLM, ZUNA), two benchmark studies (EEG-FM-Bench, Xiong et al., 2025; EEG-Bench, Kastrati et al., 2025), one systematic evaluation ("Are EEG Foundation Models Worth It?", Yang, L. et al., 2026, ICLR 2026), and one critical survey (Kuruppu et al., 2025). Per-model summaries are provided in Appendix A.

<h4>2.4.1 Taxonomy</h4>

We organize the landscape along four dimensions.

<strong>Architectural backbone.</strong> Transformers dominate (LaBraM, EEGPT, CBraMod, BIOT), with emerging alternatives in state-space models (EEGMamba, bidirectional Mamba with linear O(n) complexity), diffusion autoencoders (ZUNA, 380M parameters trained on 208 datasets), and multimodal LLM hybrids (NeuroLM, GPT-2 backbone with VQ-tokenized EEG input, up to 1.7B parameters). BENDr uses a hybrid CNN encoder feeding into a transformer. No pure CNN-based EEG foundation model exists.

<strong>Pretraining objective.</strong> Six strategies have been explored: contrastive learning (BENDr, BIOT – the earliest approach, now largely superseded), masked reconstruction with raw signal targets (CBraMod, EEGMamba), masked reconstruction with discrete VQ-code targets (LaBraM, inspired by BEiT v2), masked reconstruction with representation alignment targets (EEGPT, which addresses low EEG signal-to-noise ratio by predicting high-SNR reference representations), autoregressive next-token prediction (NeuroLM), and diffusion denoising (ZUNA). The field has converged heavily on masked reconstruction variants: seven of ten models surveyed (including the critical survey's broader count) use some form of masked reconstruction with a transformer backbone, adopted from NLP without controlled evidence that this paradigm is optimal for EEG.

<strong>Scale.</strong> Parameter counts span three orders of magnitude (BIOT at 3.3M to NeuroLM-XL at 1.7B). Pretraining data ranges from BENDr's approximately 1,500 hours to ZUNA's approximately 2 million channel-hours from 208 datasets. A critical finding, discussed below, is that neither parameter scale nor data scale reliably predicts downstream performance.

<strong>Tokenization and channel encoding.</strong> How raw EEG signals are converted into model input tokens is a fundamental design choice that determines how the model handles channel heterogeneity – different datasets using different numbers of electrodes in different spatial configurations. Approaches have evolved through four generations: fixed channel assumptions (BENDr) that fail catastrophically on mismatched layouts; discrete learnable embeddings (LaBraM, BIOT) that work for seen configurations but cannot generalize; adaptive positional encoding (CBraMod, EEGPT); and geometric 3D coordinate encoding (ZUNA's 4D-RoPE), which encodes the physical spatial coordinates of each electrode and enables generalization to arbitrary configurations including positions never seen during training. EEG-Bench found channel mismatch to be "devastating" for models with fixed channel assumptions.

<h4>2.4.2 Model Summary</h4>

<table class="dd-table">
<thead><tr><th>Model</th><th>Year</th><th>Architecture</th><th>Pretraining</th><th>Parameters</th><th>Independent Evals</th></tr></thead>
<tbody>
<tr><td>BENDr</td><td>2021</td><td>CNN + Transformer</td><td>Contrastive</td><td>~4M</td><td>3</td></tr>
<tr><td>BIOT</td><td>2023</td><td>Linear Transformer</td><td>Contrastive</td><td>3.3M</td><td>2</td></tr>
<tr><td>LaBraM</td><td>2024</td><td>ViT</td><td>Masked (VQ codes)</td><td>5.8M–369M</td><td>3</td></tr>
<tr><td>EEGPT</td><td>2024</td><td>ViT</td><td>Masked (repr. align)</td><td>~10M</td><td>2</td></tr>
<tr><td>CBraMod</td><td>2025</td><td>Criss-cross Transformer</td><td>Masked (raw)</td><td>~5M</td><td>2</td></tr>
<tr><td>EEGMamba</td><td>2025</td><td>Bidirectional Mamba</td><td>Masked (raw)</td><td>—</td><td>0</td></tr>
<tr><td>NeuroLM</td><td>2024</td><td>GPT-2</td><td>Autoregressive</td><td>254M–1.7B</td><td>0</td></tr>
<tr><td>ZUNA</td><td>2026</td><td>Diffusion Autoencoder</td><td>Diffusion denoising</td><td>380M</td><td>0</td></tr>
</tbody>
</table>

Three models (EEGMamba, NeuroLM, ZUNA) have zero independent benchmark evaluation. Only BENDr and LaBraM appear in all three benchmark studies. Self-reported results are systematically optimistic: authors select evaluation protocols and datasets that favor their models. The field's conclusions about relative model effectiveness are therefore built on a partial and potentially biased evidence base.

<h4>2.4.3 The Representation Quality Problem</h4>

The most robust negative finding across all evaluations is the frozen-backbone collapse. When model weights are frozen and only a linear classifier is trained on top, performance drops to near-chance across all models and all pretraining strategies. This renders the whole approach powerless in obtaining the goals of an emergent system with superintelligent interaction skill:

<ul>
<li>Frozen BENDr on SEED: 33.33% balanced accuracy (random chance for 3-class)</li>
<li>Linear probing is "consistently weak across ALL foundation models" (Yang, L. et al., 2026)</li>
<li>"Linear probing consistently underperforms fine-tuning, questioning whether current pretraining truly learns transferable high-level representations" (Kuruppu et al., 2025)</li>
</ul>

In NLP and computer vision, frozen features from foundation models are often sufficient for strong downstream performance, the defining property that makes foundation models useful. The universal failure of frozen EEG representations suggests something fundamentally different about what current models learn. Three explanations are plausible: (1) EEG has lower information density than text or images, and reconstruction objectives may learn to reproduce noise as faithfully as signal; (2) current objectives learn low-level spectral features rather than the abstract representations needed for downstream tasks; (3) pretraining data lacks diversity, with five of eight models pretraining primarily on Temple University Hospital EEG Corpus (TUEG) data.

We note that the magnitude of the collapse may be partly inflated by evaluation methodology. EEG-FM-Bench found that replacing simple linear probes with larger MLPs improved CBraMod by more than 10%, and EEGPT claims successful linear probing using a different protocol than the benchmarks. The qualitative finding – that frozen EEG representations are far weaker than their NLP/vision counterparts – is nonetheless robust.

<h4>2.4.4 Architecture and Objective Design Dominate Scale</h4>

Our most important comparative finding is that architecture and pretraining objective design dominate model scale as predictors of downstream performance.

<strong>The CBraMod-NeuroLM inversion.</strong> CBraMod at approximately 5M parameters matches or exceeds NeuroLM-XL at 1.7B on shared benchmarks – a 340× parameter disadvantage overcome by better architecture and pretraining design. NeuroLM-XL also underperforms LaBraM-Huge (369M parameters, approximately 2,500 hours of data) on TUAB abnormal detection (0.797 vs. 0.826 balanced accuracy) and TUEV event classification (0.468 vs. 0.662) despite having 4.6× more parameters and approximately 10× more pretraining data. These results, from the same first author (Wei-Bang Jiang), constitute the strongest evidence that the LLM-based approach sacrifices per-task performance for multimodal flexibility.

<strong>Within-architecture scaling works.</strong> LaBraM demonstrates clear improvement from Base (5.8M) to Large (46M) to Huge (369M). The anti-scaling evidence comes from cross-model comparisons where architecture and objective differ simultaneously.

<strong>A simple baseline matches complex models.</strong> The "Worth It?" paper's deliberately simple ViT + MAE baseline (ST-EEGFormer) matches elaborate foundation models, suggesting that architectural complexity in pretraining may not be necessary.

The practical implication is that the NLP playbook of "scale solves everything" does not apply to EEG. This finding directly informs RLbF's training pipeline design: the cognitive state transition predictor (Section 5.2) should prioritize design over scale.

<h4>2.4.5 Classical Methods Remain Competitive</h4>

EEG-Bench (Kastrati et al., 2025), the only dedicated clinical benchmark, reveals a striking pattern:

<table class="dd-table">
<thead><tr><th>Task</th><th>Best Classical</th><th>Best FM</th><th>Winner</th></tr></thead>
<tbody>
<tr><td>Abnormal EEG</td><td>SVM: 0.722</td><td>LaBraM: 0.838</td><td>FM</td></tr>
<tr><td>Epilepsy</td><td>LDA: 0.531</td><td>BENDr: 0.740</td><td>FM</td></tr>
<tr><td>mTBI</td><td>LDA: 0.813</td><td>LaBraM: 0.740</td><td>Classical</td></tr>
<tr><td>Schizophrenia</td><td>SVM: 0.679</td><td>Neuro-GPT<sup><a href="#fn1">1</a></sup>: 0.545</td><td>Classical</td></tr>
<tr><td>Sleep Staging</td><td>LDA: 0.671</td><td>LaBraM: 0.192</td><td>Classical</td></tr>
</tbody>
</table>

Foundation models win on tasks with large, balanced datasets from TUEG-like sources (abnormal detection, epilepsy), while classical methods with expert-engineered features (Common Spatial Patterns, neuroscience-informed spectral features) win on tasks with small samples, class imbalance, or non-TUEG data. This pattern is consistent with TUEG pretraining bias and with current pretraining failing to capture the domain knowledge encoded in decades of neuroscience research.

<h4>2.4.6 TUEG Dependency and Evaluation Fragmentation</h4>

Five of eight core models pretrain on TUEG data, and two of the most common benchmarks (TUAB, TUEV) derive from TUEG. This creates a hidden circularity: models are evaluated on data distributions they were pretrained on, making it impossible to separate pretraining quality from data familiarity. Only LaBraM (20 diverse datasets) and ZUNA (208 datasets) achieve genuine data diversity.

Evaluation protocols are not standardized across papers. Preprocessing pipelines, data splits, metric computation, and fine-tuning procedures all vary. EEG-FM-Bench found that classifier head architecture alone can change performance by more than 10%. Cross-paper performance numbers cannot be directly compared.

<h4>2.4.7 Implications for RLbF</h4>

These findings matter for RLbF because any RLbF system depends on an upstream cognitive-state decoder operating under the same constraints of heterogeneity, data bias, and limited transfer. RLbF therefore treats decoding as a separate component: in the near term, a specialized calibrated model may be sufficient; in the longer term, stronger EEG foundation models may become attractive backbones for these decoders. The immediate point is to use decoded cognitive-state estimates as a noisy continuous reward for language-model adaptation.

The survey also suggests a longer-term opportunity. If RLbF deployments generate large corpora of aligned EEG traces, decoder outputs, context windows, and outcomes from real interaction, they may eventually supply the kind of ecologically valid data that current EEG modeling lacks for interactive settings. This possibility is prospective rather than demonstrated here. Separately, the EEG foundation model field still needs controlled ablation of pretraining objectives, systematic representation diagnostics, unified clinical evaluation protocols, and exploration of under-tested architectures (state-space models for long recordings, diffusion representations for classification).

<h3>2.5 Comparison Summary and the Research Gap</h3>

The following comparison captures the key dimensions along which RLbF differs from prior approaches:

<table class="dd-table">
<thead><tr><th>Approach</th><th>Feedback Type</th><th>Temporal Resolution</th><th>Adaptation Mechanism</th><th>Generative?</th></tr></thead>
<tbody>
<tr><td>RLHF (Ouyang et al., 2022)</td><td>Discrete preferences (voluntary)</td><td>Per-response</td><td>Post-training via reward model + PPO/DPO</td><td>Yes (LLM)</td></tr>
<tr><td>RLAIF (Bai et al., 2022)</td><td>AI preferences (synthetic)</td><td>Per-response</td><td>Post-training via AI reward model</td><td>Yes (LLM)</td></tr>
<tr><td>Affective tutoring (D'Mello and Graesser, 2012)</td><td>Facial expression, dialogue cues</td><td>Per-interaction (~seconds)</td><td>Rule-based strategy selection</td><td>No</td></tr>
<tr><td>Neurofeedback (Sitaram et al., 2017)</td><td>EEG/fMRI (involuntary)</td><td>Continuous (~sub-second)</td><td>Stimulus parameter adjustment</td><td>No</td></tr>
<tr><td>Content modulation (Furman and Kwalwasser, 2021)</td><td>Decoded EEG cognitive states</td><td>Continuous (~sub-second)</td><td>Content parameter modulation</td><td>No</td></tr>
<tr><td>Empathetic dialogue (Rashkin et al., 2019)</td><td>Text cues (voluntary)</td><td>Per-response</td><td>Supervised training on empathetic labels</td><td>Yes (LLM)</td></tr>
<tr><td><strong>RLbF (this work)</strong></td><td>Decoded EEG cognitive states</td><td>Continuous (~sub-second)</td><td>Post-training via neurophysiological reward</td><td>Yes (LLM)</td></tr>
</tbody>
</table>

No existing approach combines (a) generative language models, (b) continuous involuntary neurophysiological feedback, and (c) post-training rather than rule-based adaptation. The empathetic dialogue row illustrates that text-only approaches can train generative models for empathic communication, but rely on voluntary text cues rather than involuntary neurophysiological signals: they learn to respond to what users say about their feelings, not to how users actually feel. This comparison highlights dimensions favorable to RLbF; on other dimensions – hardware requirement, training population size and diversity, empirical validation, and content quality training – RLHF is clearly superior (see Section 2.1). RLbF synthesizes elements from each prior approach that no single existing method integrates.

<h2>3. The Reinforcement Learning from Brain Feedback (RLbF) Framework</h2>

In this section we formalize the RLbF framework: the cognitive state space, the feedback loop, the temporal delay model, and the reward function. We've designed this framework to be empirically testable independently and in a variety of forms. Accordingly, all notation is defined with each component generating predictions that can be validated or disproven directly by data.

<h3>3.1 Cognitive State Space</h3>

We define the cognitive state at time <em>t</em> as a vector <strong>s</strong><sub><em>t</em></sub> = (<em>s</em><sub><em>t</em></sub><sup>(1)</sup>, …, <em>s</em><sub><em>t</em></sub><sup>(<em>d</em>)</sup>) ∈ [0,1]<sup><em>d</em></sup>, where each dimension represents a distinct cognitive construct normalized to the unit interval. We define <em>d</em> = 5 dimensions based on established EEG correlates and real-time decodability: enjoyment (frontal alpha asymmetry — the difference in alpha-band (8–12 Hz) EEG power between left and right frontal regions, associated with approach motivation; Davidson, 1992), cognitive workload (frontal theta power — theta-band (4–8 Hz) EEG power over frontal regions, which increases with mental effort; Gevins et al., 1997), auditory focus (cortical tracking — the synchronization of neural activity to the temporal structure of attended speech; Ding and Simon, 2012; Mesgarani and Chang, 2012), flow state (increased frontal theta with moderate frontocentral alpha; Katahira et al., 2018; Csikszentmihalyi, 1990), and stress/arousal (elevated beta power — increased power in the beta band (13–30 Hz), associated with arousal and anxiety; Al-Shargie et al., 2016). The continuous [0,1] representation preserves gradient information essential for reward computation and the set is robustly extendable to additional dimensions.

Cognitive state measurements are subject to both inter-individual variation (handled by per-user calibration) and intra-individual drift from fatigue and habituation. The framework therefore requires some form of baseline normalization that separates slow drift from utterance-related changes: <strong>s</strong><sub>response</sub>(τ) = <strong>s</strong>(τ) − <strong>s</strong><sub>drift</sub>(τ), where <strong>s</strong><sub>drift</sub>(τ) is an estimate of the slowly varying baseline. The practical severity of this challenge is substantial: frontal theta changes of 0.2–0.4 on the [0,1] scale over one-hour sessions due to fatigue drift alone (Gevins et al., 1997) may exceed per-utterance cognitive responses of 0.05–0.15 (estimated), making robust detrending essential. We note that linear subtraction of this kind is an idealized approximation; robustly solving the resulting covariate shift and disentangling transient event-related potentials from this baseline remains a critical open challenge for the field, which we detail fully in Section 9.3.

<h3>3.2 The Feedback Loop and POMDP Formulation</h3>

At each turn <em>t</em>, the agent observes (<em>c</em><sub><em>t</em></sub>, <strong>s</strong><sub><em>t</em></sub>) (conversation context and cognitive state), generates utterance <em>u</em><sub><em>t</em></sub> ∼ π<sub>θ</sub>(<em>u</em> | <em>c</em><sub><em>t</em></sub>, <strong>s</strong><sub><em>t</em></sub>), delivers it, waits through a temporal delay δ<sub><em>t</em></sub>, measures the post-utterance cognitive state <strong>s</strong><sub><em>t</em>+1</sub>, and computes reward. This defines a Partially Observable Markov Decision Process (POMDP): the cognitive state vector is a low-dimensional projection of the user's full internal state, and the agent acts under uncertainty about unobserved dimensions.

The temporal delay is modeled as:

<Figure src="/deep-dives/pics/deep-dives/rlbf_eq1.webp" alt="Equation 1: delta_t equals delta_base of modality plus sigma_i times length of u_t plus delta_neural." caption={"<strong>Equation (1).</strong> Temporal delay model: δ<sub>t</sub> = δ<sub>base</sub>(modality) + σ<sub>i</sub> · len(u<sub>t</sub>) + δ<sub>neural</sub>."} />

where δ<sub>base</sub> is modality-dependent, len(<em>u</em><sub><em>t</em></sub>) is measured in tokens, σ<sub><em>i</em></sub> is a per-user processing speed factor (in seconds per token, where <em>i</em> indexes the user), and δ<sub>neural</sub> is the neural processing pipeline latency (estimated at 1–3 seconds). Because cognitive responses are not instantaneous and the exact onset time is uncertain, we do not use the cognitive state at a single time point. Instead, the post-utterance cognitive state <span style="text-decoration:overline">s</span><sub><em>t</em>+1</sub> is defined as a weighted average over a Δ-second window following the estimated response onset:

<Figure src="/deep-dives/pics/deep-dives/rlbf_eq2.webp" alt="Equation 2: post-utterance cognitive state defined as a weighted integral over a delta-second window." caption={"<strong>Equation (2).</strong> Post-utterance cognitive state as a weighted temporal average over the Δ-second response window."} />

where <strong>s</strong>(τ) is the continuous cognitive state signal at time τ after utterance delivery, Δ is the window duration (a hyperparameter, typically 10 seconds), and <em>w</em>(τ; <em>t</em>) ≥ 0 is a temporal weighting function. The specific shape of <em>w</em>(τ; <em>t</em>) is an implementation choice. The choice of Δ represents a deliberate trade-off between temporal precision and noise robustness.

For streaming voice mode, where overlapping utterances create entangled cognitive responses, we recommend paragraph-level reward assignment as the practical starting point, with per-sentence deconvolution as a research direction. Text-based turn-by-turn interaction uses the per-turn model directly.

The optimization objective is the undiscounted finite-horizon return over a conversation session: we sum rewards across turns without a discount factor, because conversations are finite interactions where later turns are not inherently less valuable than earlier ones.<sup><a href="#fn2">2</a></sup>

<h3>3.3 The Three-Component Reward Function</h3>

The reward at each turn is:

<Figure src="/deep-dives/pics/deep-dives/rlbf_eq3.webp" alt="Equation 3: R_t equals weighted sum of R_pred, R_res, and R_app components." caption={"<strong>Equation (3).</strong> Three-component reward: <em>R</em><sub>t</sub> = <em>w</em><sub>pred</sub> · <em>R</em><sub>pred,t</sub> + <em>w</em><sub>res</sub> · <em>R</em><sub>res,t</sub> + <em>w</em><sub>app</sub> · <em>R</em><sub>app,t</sub>."} />

subject to <em>w</em><sub>pred</sub> + <em>w</em><sub>res</sub> + <em>w</em><sub>app</sub> = 1, <em>w</em><sub>pred</sub>, <em>w</em><sub>res</sub>, <em>w</em><sub>app</sub> ≥ 0.

<strong>Prediction accuracy (<em>R</em><sub>pred</sub>)</strong> rewards the agent for accurately predicting cognitive state transitions. A learned predictor <em>f</em><sub>ϕ</sub> maps the current context, cognitive state, and utterance to a predicted next cognitive state <strong>ŝ</strong><sub><em>t</em>+1</sub> = <em>f</em><sub>ϕ</sub>(<em>c</em><sub><em>t</em></sub>, <strong>s</strong><sub><em>t</em></sub>, <em>u</em><sub><em>t</em></sub>):

<Figure src="/deep-dives/pics/deep-dives/rlbf_eq4.webp" alt="Equation 4: R_pred equals negative squared L2 norm of difference between predicted and observed next cognitive state." caption={"<strong>Equation (4).</strong> Prediction-accuracy reward: <em>R</em><sub>pred,t</sub> = −‖<strong>ŝ</strong><sub>t+1</sub> − <span style='text-decoration:overline'>s</span><sub>t+1</sub>‖<sub>2</sub><sup>2</sup>."} />

This trains a computational analogue of empathy – an internal model of how words affect the listener. We note a subtle perverse incentive: the model could maximize prediction accuracy by steering toward predictable interactions. The resonance reward and Kullback–Leibler (KL) divergence penalties counteract this tendency, but monitoring for predictability-seeking behavior during training is recommended. As an additional countermeasure, we recommend normalizing <em>R</em><sub>pred</sub> by prediction difficulty – dividing by the variance of cognitive state transitions observed in similar conversational contexts – so that easy-to-predict interactions do not yield disproportionately high reward. This difficulty normalization should be treated as a recommended component of the Phase 3 training protocol rather than an optional enhancement.

<strong>Cognitive resonance (<em>R</em><sub>res</sub>)</strong> rewards communication style adaptation through two concrete operationalizations. <em>Information density matching</em> reduces information density (measured as mean surprisal under a fixed reference language model, normalized to [0,1]) when workload is high, according to:

<Figure src="/deep-dives/pics/deep-dives/rlbf_eq5.webp" alt="Equation 5: information density target as product of workload, focus, and stress modulators." caption={"<strong>Equation (5).</strong> Information-density target function combining workload, focus, and stress-modulated terms."} />

where <em>s</em><sup>(wkl)</sup>, <em>s</em><sup>(foc)</sup>, <em>s</em><sup>(str)</sup> denote the workload, auditory focus, and stress dimensions of the cognitive state vector (Section 3.1), and λ<sub>stress</sub> ∈ [0,1] is a stress-workload interaction modifier that distinguishes productive challenge from overwhelm. <em>Emotional tone alignment</em> targets a warmth level that increases when stress is high and enjoyment is low:

<Figure src="/deep-dives/pics/deep-dives/rlbf_eq5a.webp" alt="Equation 5a: warmth target as (1 + stress − enjoyment) / 2." caption={"<strong>Equation (5a).</strong> Warmth target: <em>g</em><sup>(warmth)</sup>(<em>s</em><sup>(enj)</sup>, <em>s</em><sup>(str)</sup>) = (1 + <em>s</em><sup>(str)</sup> − <em>s</em><sup>(enj)</sup>) / 2."} />

where <em>s</em><sup>(enj)</sup> denotes the enjoyment dimension. When stress exceeds enjoyment, the target warmth rises above 0.5; during positive engagement (high enjoyment, low stress), the target drops, allowing the model to maintain stylistic stability during flow states. Both operationalizations are presented as testable hypotheses: the specific target functions are starting points grounded in Cognitive Load Theory (Sweller, 1988; Bjork and Bjork, 2011), attentional control theory (Eysenck et al., 2007), and therapeutic alliance research (Horvath and Symonds, 1991; Barrett-Lennard, 1962), not settled definitions. The CLT grounding extends to the information density target itself: the "desirable difficulties" framework (Bjork and Bjork, 2011) predicts that moderate-to-high information density is beneficial when cognitive capacity permits, supporting the non-zero target implied by Eq. 5 when workload is low. We also note that the stress-workload interaction modifier λ<sub>stress</sub> is the only cross-dimension interaction currently specified; whether additional interactions (e.g., between enjoyment and focus, or between flow and workload) improve the resonance reward remains an empirical question for future work on the joint target function design. A learned joint target function is the principled long-term solution.

<strong>Application-layer reward (<em>R</em><sub>app</sub>)</strong> is an optional, context-dependent term for deployment-specific objectives (learning gain in tutoring, distress reduction in therapeutic support). The default is <em>w</em><sub>app</sub> = 0: pure empathic attunement with no externally specified goal.

The complete constrained reward (Eq. 6) extends the three-component reward (Eq. 3) with diversity, KL-divergence, and prosodic diversity penalties to prevent reward hacking (exploiting loopholes in the reward function to achieve high scores without the intended behavior) — including convergence on rhythmic patterns that entrain neural oscillations (Giraud and Poeppel, 2012), a concrete vulnerability given the auditory focus dimension's sensitivity to prosodic structure.

The three-component reward design reflects RLbF's structural differences from existing alignment methods. Of the eleven dimensions in which RLbF and RLHF diverge, the three limitations identified in Section 1.2 are the most fundamental: RLbF increases feedback density by orders of magnitude (1+ Hz versus one label per response) and replaces population-level aggregation with individual specificity. As discussed in Section 2.1, RLbF is complementary to RLHF, not a replacement: RLHF trains for content quality through socially constructed preferences that require deliberate judgment, while RLbF trains for communication style adaptation through involuntary cognitive signals.

With the mathematical framework established, the next step is its concrete realization: the Isaac platform, which implements the cognitive feedback loop, tokenizes the state space for LLM consumption, and supports both closed-loop and open-loop inference.

<h2>4. System Implementation: The Isaac Platform</h2>

<h3>4.1 The Isaac Architecture</h3>

To realize the RLbF POMDP described in Section 3, we built Isaac, a platform that maps the abstract theoretical variables to a live software architecture. The system implements a closed loop: the user's cognitive state, measured via EEG and decoded by Arctop's on-device pipeline, feeds back into the language model's context, shaping every subsequent utterance. Seven components form the cycle: the user's brain produces neural activity; EEG hardware acquires raw signals; the Arctop on-device decoder transforms raw EEG into cognitive scores (enjoyment, workload, focus, flow, stress) emitted at 1 Hz, with raw EEG never leaving the device; a cognitive state tokenizer converts scores into a representation for the LLM; the LLM generates text conditioned on the augmented context; and a stream controller delivers text sentence by sentence, pausing between sentences to allow cognitive state updates. The sentence is the atomic unit of cognitive feedback, with an estimated minimum loop period of 3–5 seconds.

This design takes as inspiration the elegant formulations of late mathematician and natural philosopher Sir Isaac Newton, who in 1686 published his treatise on the three laws of motion the "Principia Mathematica Philosophiae Naturalis." A fundamental insight was that (law #3) for every action in nature there is an equal and opposite reaction. For our current purposes with RLbF, a corollary is proposed to this physical law: when object 'AI' exerts a force on object 'Brain,' then object 'Brain' also exerts a force on object 'AI.'

The software platform we use for exploration and testing related to this topic, 'Isaac' (Arctop Inc., 2026, v1.15. <a href="https://arctop.com/">https://arctop.com/</a>), implements a dual-agent architecture: an AI conversation agent that speaks with the human user, plus an AI recommendation agent that interprets cognitive state patterns from the human and advises the AI conversation agent, serving as a simulacrum of sorts for human neurobiologically-emergent and felt 'empathy.'

<Figure src="/deep-dives/pics/deep-dives/rlbf_figure3.webp" alt="The Isaac Framework: A Dual-Agent Partially Observable Markov Decision Process (POMDP) Architecture." caption={"<strong>Figure 3. The Isaac Framework: A Dual-Agent Partially Observable Markov Decision Process (POMDP) Architecture.</strong>"} />

The CognitiveTracker buffers scores, detects significant shifts, and computes trends. The RecommendationAgent generates natural language advisories (e.g., "Workload has risen sharply. Consider simplifying your next response."). The StreamController manages sentence-level delivery. The SessionRecorder captures timestamped interaction data for RLbF training. This dual-agent pattern is a deliberate stepping stone and future versions follow several distinct branches to help flesh out the best architectures for this structure of contextual data.

It is critical to distinguish Isaac's current proof-of-concept architecture from the final RLbF-trained model. The dual-agent structure is a prompt-based scaffolding designed to bypass the need for architectural modifications to the base LLM during initial data collection. By translating continuous cognitive scores into natural language advisories, the Recommendation Agent allows an off-the-shelf instruction-tuned model to simulate empathic adaptation. However, this is a transitional architecture. In the fully realized Phase 3 RLbF model, the policy is optimized directly against the mathematical, three-component reward (<em>R</em><sub>total</sub>). At that stage, the LLM internalizes the cognitive state mappings into its weights, rendering the Recommendation Agent obsolete for real-time inference and resulting in a lower-latency, single-agent system.

<h3>4.2 Signal Processing Engineering</h3>

Isaac's signal-processing layer instantiates the abstract requirements defined in Section 3 with specific, modular engineering choices. For the baseline normalization in Section 3.1, Isaac estimates the slowly varying baseline <strong>s</strong><sub>drift</sub>(τ) as a 60-second moving average of the cognitive state signal, subtracted from the raw scores before reward computation and predictor training. This window is long enough to average out utterance-scale fluctuations but short enough to track fatigue and habituation drift. For the temporal integration weighting <em>w</em>(τ; <em>t</em>) in Eq. 2, Isaac uses a trapezoidal window with ramp-up and ramp-down periods, which avoids sharp boundary artifacts while remaining simple to implement; a truncated Gaussian centered at δ<sub><em>t</em></sub> + Δ/2 is a principled alternative. Both choices are modular: the framework accommodates different baseline estimators and window shapes, and Section 9.3 details known limitations of linear-subtraction detrending that motivate more sophisticated transfer-learning approaches in future work.

<h3>4.3 Cognitive Tokenization via Prompt Injection</h3>

We recommend natural language injection as the primary tokenization strategy: cognitive state is injected as a natural language string (e.g., "Cognitive workload: 0.85 (high – approaching overload)") into the LLM's context between sentences. This requires no architectural modifications, preserves full numerical precision, supports transparency (the injected text is human-readable for auditing), and works with any instruction-tuned model. Learned embeddings are the principled long-term optimization for production systems, but natural language injection is preferred for the research phase.

<h3>4.4 Voice Mode vs. Text Mode</h3>

The architecture supports two delivery modalities with different temporal dynamics. Voice mode introduces TTS latency (200–800ms per sentence), fixed delivery rate (~150 words/min), and auditory processing dynamics where the auditory focus dimension is directly informative. Text mode eliminates TTS latency but introduces variable reading speed (200–400 words/min) and self-paced processing, making temporal alignment less precise. The LLM uses the same weights for both modalities; modality-specific adaptation is learned during RLbF training from interaction data, not hard-coded. A modality parameter affects the cognitive state tokenizer (labeling which dimensions are relevant), the stream controller (switching delivery mechanism), and the delay model parameters.

<h3>4.5 Individual Calibration</h3>

Users differ in baseline cognitive state patterns, cognitive responses to language, and processing speed. A brief calibration session (estimated at 5–10 minutes) at first interaction establishes individual baselines (resting-state measurement), response characteristics (controlled stimuli eliciting known cognitive patterns), and conversational processing speed. Two adaptation strategies are supported: a few-shot cognitive profile injected as natural language into the system prompt (default for new users), and lightweight parameter adaptation (LoRA – Low-Rank Adaptation; Hu et al., 2022 – of the transition predictor) for returning users with accumulated data.

<h3>4.6 Inference Modes</h3>

Both closed-loop (with EEG) and open-loop (without EEG) modes use identical model weights. The only difference is the source of cognitive state: observed scores from the Arctop decoder in closed-loop mode, or predicted scores from the internal transition predictor <em>f</em><sub>ϕ</sub> in open-loop mode. The cognitive state tokenizer checks whether an Arctop score stream is available; if yes, it uses observed scores; if no, it queries the internal predictor. The model does not know which source is providing the scores – it receives the same natural language injection format in both cases. This design enables smooth transitions between modes, including mid-session device removal, and makes trained empathic capabilities available to users without EEG hardware.

The Isaac platform described above provides the runtime infrastructure; what remains is the training procedure that transforms a standard instruction-tuned LLM into a model capable of exploiting this infrastructure for empathic communication.

<h2>5. Training Pipeline</h2>

The pipeline is designed to transform a standard instruction-tuned LLM into a model that natively understands and responds to real-time cognitive state signals through three phases of increasing complexity. Phases 1 and 2 use Isaac's prompt-injected dual-agent scaffolding (Section 4) to generate and collect training data; Phase 3 then trains a single pure RLbF policy that replaces the scaffolding and natively optimizes against the reward math defined in Section 3.

<h3>5.1 Phase 1: Supervised Fine-Tuning for Cognitive State Comprehension</h3>

Phase 1 teaches the model to parse cognitive state injections and adjust communication style accordingly. This phase is necessary because standard instruction-tuned LLMs have no prior exposure to cognitive state tokens; without supervised grounding, the model would treat injected scores as noise rather than actionable context, undermining the reward signal in subsequent phases.

Synthetic data is generated through three layers. Template-based generation defines cognitive state profiles (overwhelmed, productively challenged, bored, in-flow, stressed-but-coping, relaxed) with expected communication adaptations grounded in the resonance reward's target functions. LLM-augmented variation expands coverage to thousands of diverse (conversation, cognitive_state, response) triples, ensuring that the model encounters cognitive state signals across a broad range of topics, registers, and conversational depths. Isaac advisory bootstrapping grounds the synthetic distribution in real cognitive state patterns from existing session logs, mitigating the risk that purely synthetic data teaches the model to respond to cognitive state profiles that do not occur in practice.

Success criteria include cognitive state parsing accuracy (&gt;85% agreement with an LLM judge), measurable style adaptation (information density decreasing by at least 15% under high workload), appropriate comprehension check insertion, and no degradation of general conversational quality (within 5% on standard benchmarks). Estimated data requirements: 15,000–25,000 training examples (these figures are preliminary estimates pending empirical calibration).

<h3>5.2 Phase 2: Prediction Calibration with Real EEG Data</h3>

Phase 2 solves the hardest technical challenge: learning to predict how utterances affect cognitive state from real Isaac session recordings. This is the phase that grounds the framework in empirical reality – the transition from synthetic cognitive state profiles (Phase 1) to measured human cognitive responses. Its success determines whether the prediction accuracy reward (<em>R</em><sub>pred</sub>) and the open-loop inference mode (Section 4.6) are feasible: if the predictor cannot learn meaningful dynamics from real data, neither the reward signal nor the open-loop transfer hypothesis has a foundation.

An extraction pipeline processes raw recordings through sentence-level segmentation, drift detrending (60-second moving average subtraction), pre-utterance state measurement, post-utterance windowed aggregation, and quality filtering, producing aligned (<em>c</em><sub>t</sub>, <strong>s</strong><sub>t</sub>, <em>u</em><sub>t</sub>, <span style={{textDecoration:'overline'}}>s</span><sub>t+1</sub>) tuples. Conservative temporal attribution truncates observation windows at the onset of subsequent utterances to prevent cross-sentence contamination. Quality filtering discards turns where EEG signal quality falls below a per-channel threshold or where the temporal gap between utterance delivery and state measurement falls outside the delay model bounds (Eq. 1), ensuring that the training set contains only reliably attributed cognitive state transitions.

The transition predictor <em>f</em><sub>ϕ</sub> is trained on these tuples with population-level user embeddings that capture individual variation within a single model. Success criteria include per-dimension prediction correlation exceeding 0.3 for at least 3 of 5 dimensions on held-out users, outperforming the no-change baseline by at least 15%, and an autoregressive open-loop horizon exceeding 10 turns. We estimate the minimum data requirements to perform this sufficiently to be: 250–500 sessions from 30 unique and unrelated EEG-equipped users.

A significant challenge in predicting cognitive state transitions from text is the problem of temporal credit assignment: isolating which specific tokens within an utterance evoked which specific neurological response. Human cognitive states do not shift in response to structural filler words (e.g., 'the', 'is'), but rather have their own unique timescales that anchor to semantically salient or high-arousal tokens. To solve this, we leverage the natural language logs generated by the Isaac Recommendation Agent during Phase 1 data collection. The Recommendation Agent's historical advisories serve as a semantic mask during Phase 2 predictor training. By aligning the system's structural understanding of high-impact words with the recorded temporal delays, the predictor learns to ignore syntactic filler and attribute cognitive state changes specifically to the high-arousal tokens within the context window. This prevents the model from smearing the predicted cognitive response uniformly across an entire sentence.

<h3>5.3 Phase 3: Reinforcement Learning with Empathic Reward</h3>

Phase 3 is the step at which the model leaves the Isaac scaffolding behind. In Phases 1 and 2 the target model sits inside Isaac's prompt-injected dual-agent loop (Section 4.1), receiving RecommendationAgent advisories as part of its context. In Phase 3, PPO trains the model to operate as a single policy that reads the cognitive-state injection directly and natively optimizes against the reward math in Section 3, with the RecommendationAgent retained only at training time (as the semantic mask for the Phase 2 predictor, §5.2) and absent at inference.

Phase 3 optimizes the policy using Proximal Policy Optimization (PPO; Schulman et al., 2017) against the full reward function:

<Figure src="/deep-dives/pics/deep-dives/rlbf_eq6.webp" alt="Equation 6: total reward as sum of prediction, resonance, application, diversity, KL, and prosodic components." caption={"<strong>Equation (6).</strong> Full constrained reward including diversity, KL-divergence, and prosodic penalties."} />

with the Phase 1 SFT (Supervised Fine-Tuning) model as the starting and reference policy and the Phase 2 prediction model providing <em>R</em><sub>pred</sub>.

We choose PPO over the DPO family (Rafailov et al., 2023) based on a specific analysis of RLbF's requirements, not a generic preference. The DPO landscape has evolved substantially: KTO (Ethayarajh et al., 2024) eliminates the need for pairwise preference data by optimizing against a per-example utility function, online iterative DPO variants (Guo et al., 2024; Xu et al., 2024) address the original algorithm's offline limitation, and reward-weighted regression methods can handle continuous reward signals. These advances make DPO-family methods viable for a wider range of alignment problems than the original formulation allowed. Nevertheless, three properties of the RLbF setting specifically favor PPO:

First, RLbF's reward is a composite of six components (<em>R</em><sub>pred</sub>, <em>R</em><sub>res</sub>, <em>R</em><sub>app</sub>, <em>R</em><sub>diversity</sub>, <em>R</em><sub>KL</sub>, <em>R</em><sub>prosodic</sub>) with different scales, noise characteristics, and update frequencies. PPO's learned value function (which estimates expected future reward from a given state) can internalize the relative scaling and variance of these components, enabling stable optimization against the composite signal. DPO-family methods would require collapsing this structure into a single scalar or pairwise ranking before optimization, losing information about which reward components drive each comparison.

Second, the prediction model <em>f</em><sub>ϕ</sub> continues to update during Phase 3 as the policy generates novel utterances that fall outside the Phase 2 training distribution. This creates a non-stationary reward landscape where online exploration – generating utterances, observing their predicted cognitive effects, and updating the policy – is essential. PPO's on-policy sampling (generating new training data from the current policy rather than reusing old data) naturally supports this co-evolution of policy and reward model. While online DPO variants address the static-dataset limitation of the original algorithm, they still optimize against pairwise preference rankings rather than directly against a continuous, multi-dimensional reward signal that itself evolves during training.

Third, RLbF requires credit assignment across sentence boundaries: a single utterance's cognitive impact unfolds over a temporal window (Section 3.2), and the reward signal reflects the cumulative effect of multiple conversational turns on cognitive state trajectories. PPO's value function learns to estimate long-horizon returns, enabling credit assignment (determining which earlier actions caused later outcomes) that connects current utterance choices to downstream cognitive outcomes. This temporal credit assignment problem does not map naturally onto the pairwise comparison framework that underlies DPO and its variants, including KTO's per-example formulation.

The training supports both online (live Isaac sessions with real-time EEG) and offline (replaying Phase 2 session recordings with the prediction model) modes. In offline mode, the prediction model estimates cognitive state transitions for policy-generated utterances that differ from the recorded ones, introducing distributional shift (the mismatch between the training data distribution and the data the current policy generates) that limits the extent of offline optimization. A hybrid approach – offline pre-optimization to get a reasonable policy, followed by online fine-tuning with live EEG feedback – is recommended to balance data efficiency with the correction that real interaction provides. The recommended three-stage sequence and its performance-based transition criteria are:

<ol>
<li><strong>Offline RL warm-up</strong> (1–2 epochs over Phase 2 recordings): Establishes a baseline policy using conservative reward estimation.</li>
<li><strong>Hybrid online/offline</strong> (iterative): Short online sessions (15–30 minutes) with EEG-equipped users, combined with continued offline training. Gradually shifts the data distribution toward on-policy.</li>
<li><strong>Full online RL</strong>: Standard PPO training with live interaction for final optimization.</li>
</ol>

<strong>Transition criterion: offline to hybrid.</strong> Advance from offline-only to hybrid training when the offline-trained policy achieves <em>R</em><sub>res</sub> ≥ 1.10 × <em>R</em><sub>res</sub><sup>SFT</sup> (at least 10% improvement in resonance reward over the Phase 1 SFT baseline) on a held-out set of Phase 2 recordings, AND the mean estimated reward <em>R̂</em><sup>total</sup> has plateaued (less than 2% relative improvement over the preceding 500 gradient steps). The first condition ensures the policy has learned nontrivially from offline data; the second ensures that offline returns are saturating and further improvement requires on-policy data.

<strong>Transition criterion: hybrid to full online.</strong> Advance from hybrid to full online training when two conditions hold: (1) the per-dimension prediction correlation of <em>f</em><sub>ϕ</sub> on the most recent online session batch meets or exceeds the Phase 2 baseline threshold (<em>r</em> ≥ 0.3 for at least 3 of 5 cognitive dimensions), confirming that the co-trained prediction model has adapted to the evolving policy distribution; and (2) the ratio of online-to-offline reward diverges by less than 10%, i.e., |<em>R</em><sup>online</sup> − <em>R̂</em><sup>offline</sup>| / <em>R</em><sup>online</sup> &lt; 0.10, indicating that the offline reward estimates have become unreliable enough that continued offline training adds diminishing value. At this point, the policy is stable enough for sustained live interaction and the reward signal quality justifies full online operation.

The KL divergence penalty coefficient β<sub>KL</sub> may require higher values (0.05–0.2) than typical RLHF settings because the cognitive state reward signal is noisier than human preference judgments. Training is monitored continuously for reward hacking indicators (Anthropic, 2025; Fu et al., 2025):

<ul>
<li>Prosodic convergence</li>
<li>Semantic diversity collapse</li>
<li>Complexity collapse</li>
<li>Question frequency spikes</li>
<li>Warmth saturation</li>
</ul>

The specific values of constraint hyperparameters – including β<sub>KL</sub>, the diversity penalty weight, and the prosodic diversity coefficient are not fixed a priori but are to be calibrated empirically during Phase 3 training. Initial ranges are informed by the RLHF literature and the noise characteristics of EEG-derived reward, but final values should be determined through hyperparameter search on a validation set of held-out session recordings, with the reward hacking indicators above serving as diagnostic criteria for under- or over-regularization.

<h2>6. Open-Loop Empathic Transfer</h2>

<em>Terminological note</em>: In this section, we use "agent" in the decision-theoretic context (the POMDP actor) and "model" when discussing the underlying LLM; these refer to the same system.

<h3>6.1 The Hypothesis</h3>

The open-loop transfer hypothesis determines whether RLbF is a general-purpose training methodology or a niche technique for BCI users: does the benefit of brain feedback training persist when the EEG signal is removed at inference?

Formally, letting π<sub>RLbF</sub> denote the RLbF-trained policy and π<sub>RLHF</sub> denote a standard RLHF-trained policy from the same base model, the hypothesis claims that in open-loop deployment (no EEG):

<Figure src="/deep-dives/pics/deep-dives/rlbf_eq7.webp" alt="Equation 7: expected empathic quality of RLbF policy exceeds that of RLHF policy in open-loop deployment." caption={"<strong>Equation (7).</strong> Open-loop empathic transfer: 𝔼[<em>Q</em><sub>empathic</sub>(π<sub>RLbF</sub>(c))] &gt; 𝔼[<em>Q</em><sub>empathic</sub>(π<sub>RLHF</sub>(c))]."} />

where <em>Q</em><sub>empathic</sub> is a composite empathic quality measure and 𝒟 is the evaluation task distribution. If true, RLbF becomes a general training methodology: train with brain feedback on a moderately sized population, deploy without brain feedback to all users. If false, RLbF is limited to closed-loop applications.

<strong>This claim is a hypothesis, not a demonstrated result.</strong> The experimental design below (Section 7.4) specifies the falsification criterion for Eq. 7.

<h3>6.2 Theoretical Grounding</h3>

Three independent perspectives support the hypothesis. First, the Arctop synthetic brain model (Furman and Kwalwasser, US20210390366A1) demonstrates in commercial deployment that models trained on paired (content, neural) data can predict from content features alone – the neural features serve as privileged information (Vapnik and Vashist, 2009; Lopez-Paz et al., 2016) during training. Second, RLHF generalization research shows that models trained with preference feedback generalize beyond their reward distribution (Kirk et al., 2024), and RLbF's richer reward signal (Section 1.2) may provide favorable conditions for at least comparable generalization. Third, cognitive science research on theory of mind (ToM) (Premack and Woodruff, 1978; Baron-Cohen, 1995), including simulation theory accounts of mindreading (Gallese and Goldman, 1998; Goldman, 2006), shows that skilled human communicators develop internal models of their interlocutors through rich multimodal feedback, then apply these models effectively in feedback-impoverished settings.

<h3>6.3 Two Levels of Transfer</h3>

We distinguish style transfer (Level 1, likely) from state inference (Level 2, aspirational). At Level 1, the model has internalized population-level regularities about what kinds of language produce what kinds of cognitive responses – adaptive information density, emotional calibration, engagement preservation, comprehension checking. At Level 2, the model infers specific users' cognitive states from conversational cues alone. Level 1 is the conservative claim; Level 2 requires the model to develop a genuine computational theory of mind and remains a research goal.

A caveat is warranted regarding the ToM analogy. Whether LLMs can develop genuine theory of mind – as opposed to surface-level behavioral mimicry of mentalizing – is scientifically contested (Ullman, 2023), and claims of emergent ToM in large models have faced significant methodological criticism. This debate bears directly on the plausibility of Level 2 transfer, which depends on the assumption that an LLM can learn to infer individual cognitive states from conversational cues in a manner analogous to human mentalizing. Level 1 transfer, by contrast, requires only that the model acquire statistical regularities between language patterns and population-level cognitive responses, a form of distributional learning that does not presuppose ToM. We therefore regard Level 2 as the more speculative claim in the framework and flag it as dependent on assumptions that remain open questions in the field.

<h3>6.4 The Sycophancy Counterargument</h3>

The most important counterargument is that the model has learned to be generically "nicer" rather than genuinely adaptive, producing warmer, simpler responses without sophisticated context-dependent adaptation. We identify this as "neurological sycophancy": the displacement of deliberative sycophancy to the neural level, where the model optimizes for brain pleasure signals rather than genuine communicative benefit. Five ordered diagnostic tests are designed to distinguish empathic adaptation from generic niceness:

<ol>
<li>Context-dependent complexity modulation</li>
<li>Appropriate challenge (correcting errors despite comfort cost)</li>
<li>Information density adaptation to inferred expertise</li>
<li>Emotional specificity across different negative emotions</li>
<li>Cognitive state trajectory prediction accuracy</li>
</ol>

<h3>6.5 Experimental Design</h3>

The open-loop transfer hypothesis is empirically testable through a controlled comparison of models trained with and without brain feedback, evaluated with and without EEG at inference. The experimental logic requires four conditions from the same base model: a Baseline LLM (no post-training), an RLHF-trained control (same base model fine-tuned with standard RLHF, isolating the post-training signal as the sole variable), an RLbF-trained closed-loop ceiling condition, and the critical test: an RLbF-trained model deployed open-loop (without EEG). The critical comparison is open-loop RLbF versus RLHF: if the open-loop RLbF model demonstrates superior empathic quality despite having no access to EEG at inference, this constitutes evidence that brain feedback training produces durable communication skills. Section 7.4 specifies the full experimental protocol, including the within-subjects Latin square design, sample size justification, statistical tests, and confound controls.

<h2>7. Roadmap to Empirical Validation</h2>

Our proposed evaluation framework is organized into four tracks, each targeting a different category of claims to be empirically tested. The protocol is specified at pre-registration quality: with each metric, statistical test, baseline, and sample size defined with enough detail for reproducibility. The protocol distinguishes pilot evaluations (feasible with current Isaac data, suitable for arXiv) from full evaluations (requiring dedicated recruitment, journal submission).

<h3>7.1 Track A: Prediction Accuracy</h3>

This track evaluates whether the cognitive state predictor <em>f</em><sub>ϕ</sub> learns useful dynamics of how language affects cognition. The test set comprises held-out Isaac sessions with strict user-level partitioning: no user appears in both training and test sets. The primary evaluation uses leave-one-user-out cross-validation across a minimum of 30 unique users, ensuring that prediction accuracy reflects generalization to new individuals rather than memorization of user-specific patterns.

Metrics include per-dimension mean absolute error (target: MAE &lt; 0.10 on the [0,1] scale, representing the perceptual boundary between comfortable and effortful processing, and a meaningful improvement over the expected persistence-baseline MAE of approximately 0.12), Pearson correlation between predicted and observed cognitive state transitions (target: <em>r</em> &gt; 0.5, anchored to the EEG-based affect recognition literature where models typically achieve <em>r</em> = 0.4–0.7; Koelstra et al., 2012), and change-based Δ-MAE (target: &lt; 0.08). Four baselines establish the performance floor: random prediction, population mean, persistence (predicting no change), and linear regression on hand-crafted utterance features (token count, GPT-2 surprisal, VADER sentiment). Users contributing fewer than 3 sessions are excluded from the k-fold cross-validation to ensure sufficient within-user data. An open-loop accumulation test further assesses whether prediction accuracy degrades gracefully over multi-turn conversations or exhibits compounding error drift.

<h3>7.2 Track B: Self-Report Validation</h3>

This track validates perceived empathic quality and establishes convergent validity between self-report and EEG-derived measures. Four adapted instruments capture complementary constructs: the Working Alliance Inventory (WAI-SR) measures perceived rapport and collaboration, the Interpersonal Reactivity Index empathic concern subscale (IRI-EC) measures perceived empathy, the NASA Task Load Index (NASA-TLX) measures perceived cognitive demand, and a custom 5-item RLbF Adaptation Perception Scale measures perceived responsiveness to the participant's cognitive state.

The WAI-SR and IRI-EC were originally validated for human-human therapeutic interaction (Hatcher and Gillaspy, 2006; Davis, 1983); early work adapted the WAI-SR for human-agent interaction (Bickmore et al., 2005), but psychometric properties of these instruments in modern human-AI contexts remain an open empirical question. Assessing construct validity, internal consistency, and factor structure of the adapted instruments is therefore a planned component of this study's pilot phase, rather than an assumed given. Self-report vs. EEG correlations test ecological validity across five pre-specified dimension pairs (e.g., NASA-TLX Mental Demand vs. mean session workload, target: <em>r</em> &gt; 0.30; IRI-EC vs. mean enjoyment trajectory slope). These thresholds are grounded in published EEG-self-report correlation data: NASA-TLX vs. EEG workload indices typically achieve <em>r</em> = 0.3–0.6 (Wobrock et al., 2015; Kamzanova et al., 2014), while EEG-affect vs. self-report correlations are typically lower (<em>r</em> = 0.15–0.35; Koelstra et al., 2012). Holm-Bonferroni correction is applied across the 5 convergent validity tests to control the family-wise error rate.

<h3>7.3 Track C: Cognitive Trajectory Analysis</h3>

This track tests whether RLbF-trained models produce measurably different cognitive state trajectories compared to non-adaptive baselines. Rather than evaluating single-turn responses, Track C examines the temporal dynamics of entire conversations, capturing how the model's adaptive behavior shapes cognitive experience over multi-turn interactions.

Four trajectory-level metrics are computed for each session: cognitive state stability (variance of each dimension relative to the baseline condition reference variance), recovery speed from overload and stress episodes (time to return below a per-dimension threshold after exceedance), trajectory coherence (mutual information between utterance features and subsequent cognitive state changes, estimated via the Kraskov <em>k</em>-nearest-neighbors estimator with <em>k</em> = 5; Kraskov et al., 2004), and cumulative time in the productive zone. The productive zone is defined per dimension (e.g., workload between 0.30 and 0.75, stress below 0.60) based on Cognitive Load Theory thresholds where learning and engagement are optimized.

<h3>7.4 Track D: Open-Loop Transfer A/B Test</h3>

This track provides the definitive test of the open-loop transfer hypothesis (Section 6.1, Eq. 7) through a controlled within-subjects comparison of four conditions from the same base model: Baseline LLM (no post-training), RLHF-trained (same base model fine-tuned with standard RLHF on a general preference dataset, isolating the post-training method as the sole variable), RLbF-trained closed-loop (ceiling – full EEG feedback during interaction), and RLbF-trained open-loop (the critical test – brain-feedback-trained model deployed without EEG). The EEG device actively records data in all four conditions; only the real-time data feed to the model varies (see evaluation protocol for details). To achieve the target of 60 completers (15 full Latin squares of 4 conditions), 70 participants are recruited to account for an estimated 15% attrition rate. Each participant completes four 35-minute conversational sessions across different topics drawn from a standardized pool with random assignment.

Eight pre-registered hypotheses are tested with repeated-measures ANOVA. Directional hypotheses (H1–H5, H7) are tested with one-tailed paired t-tests, as the predicted direction is specified a priori and the reverse direction is not theoretically meaningful. Hypotheses H1–H4 and H7–H8, which test closed-loop RLbF superiority over baselines, form one correction family (6 pairwise comparisons) with Holm-Bonferroni correction. The critical open-loop transfer hypothesis – that open-loop RLbF outperforms RLHF on the composite empathic quality measure – is tested as a separate, pre-registered primary hypothesis at α = 0.05 (uncorrected). This separation is justified because the open-loop comparison tests a conceptually distinct claim (whether brain-feedback training produces durable skills that transfer without EEG at inference) from the closed-loop superiority hypotheses (whether real-time EEG feedback improves communication quality). Testing it within the closed-loop family would penalize the study's most novel claim for sharing a correction family with hypotheses it neither depends on nor competes with.

The open-loop comparison is powered at 80% to detect a small-to-medium effect of <em>d</em> = 0.37 (paired t-test, α = 0.05 two-tailed, assumed within-subject correlation <em>r</em> = 0.5, yielding <em>N</em> = 60). The <em>d</em> = 0.37 target represents the minimum practically meaningful improvement over RLHF – the smallest effect that would justify the additional infrastructure cost of brain-feedback training (EEG data collection, cognitive state decoding, and the multi-phase training pipeline) over standard preference-based alignment. This threshold is grounded in three considerations. First, meta-analyses of empathic communication interventions in therapeutic and educational settings report effect sizes of <em>d</em> = 0.3–0.6 for perceived empathy differences (Horvath and Symonds, 1991; Bickmore et al., 2005), placing <em>d</em> = 0.37 at the lower bound of the range where empathic quality differences become reliably perceptible to recipients. Second, the RLHF alignment literature reports effect sizes of <em>d</em> = 0.4–0.8 for preference-trained models over base models on human evaluation metrics (Ouyang et al., 2022; Stiennon et al., 2020); a <em>d</em> = 0.37 improvement of RLbF-OL over RLHF thus represents a conservatively meaningful increment beyond standard alignment. Third, in practical terms, <em>d</em> = 0.37 on the productive time proportion scale corresponds to approximately 4–5 additional minutes per 35-minute session spent in the cognitive productive zone – a difference large enough to have pedagogical or therapeutic significance in sustained conversational interaction. To assess sensitivity to this assumption, Figure S1 in the supplementary materials reports a power curve showing the required <em>N</em> for effect sizes ranging from <em>d</em> = 0.2 to <em>d</em> = 0.5: at <em>d</em> = 0.2, <em>N</em> = 199 would be required; at <em>d</em> = 0.3, <em>N</em> = 90; at <em>d</em> = 0.5, <em>N</em> = 34. The chosen <em>N</em> = 60 provides 80% power at <em>d</em> = 0.37 and retains 65% power even if the true effect is as small as <em>d</em> = 0.30.

TOST (Two One-Sided Tests) equivalence testing determines whether open-loop performance falls within a pre-specified equivalence bound (Δ = 0.10 on productive time proportion) of the closed-loop ceiling. The Δ = 0.10 margin represents the meaningful-difference boundary: it is approximately half the expected RLbF-CL advantage over BASE, following the standard practice of setting equivalence margins at half the expected treatment effect (Lakens, 2017), and a degradation beyond this threshold would place open-loop performance below the expected RLHF level, negating the practical case for brain-feedback training. Measures span blind human evaluation of response quality, cognitive metrics from EEG (collected in all conditions but fed to the model only in the closed-loop condition), and participant self-report using the instruments from Track B. Bayes factors (BF<sub>10</sub>) are reported for all significant results using the default JZS prior with scale parameter <em>r</em> = 0.707 (Rouder et al., 2009). Confounds are addressed through counterbalancing (novelty effects), sham EEG with active recording in all conditions (Hawthorne effects), EEG signal quality covariates, the standardized topic pool (topic effects), restricted scheduling within a 10:00–16:00 window with within-participant time consistency (fatigue and circadian effects), automated condition assignment with experimenter blinding (experimenter bias), and a post-study manipulation check assessing whether participants identified the adaptive condition (demand characteristics).

The evaluation protocol specifies how to measure RLbF's claims; equally important is whether those claims should be pursued at all. Deploying a system that monitors and responds to users' cognitive states raises ethical questions that must be addressed as part of the framework design, not deferred to post-deployment review.

<h2>8. Ethical Framework</h2>

Our ethical analysis is positioned within two established frameworks for neurotechnology governance.

Yuste et al. (2017) proposed four ethical priorities for neurotechnologies and AI (privacy, identity, agency, and equality) in a landmark Nature commentary from the Morningside Group. RLbF engages three directly: privacy through on-device data minimization (Section 8.3), agency through user control safeguards and the empathic/persuasive distinction (Sections 8.1, 8.4), and identity through autonomy protections against long-term communicative shaping. The equality priority – ensuring cognitive enhancement technologies do not exacerbate existing inequalities – remains an open policy challenge not addressed by architectural safeguards alone.

Independently, Ienca and Andorno (2017) articulated four fundamental 'neurorights' to protect individuals' mental processes from unauthorized access, manipulation, or discrimination by neurotechnology: cognitive liberty, mental privacy, mental integrity, and psychological continuity. These rights have since influenced Chile's constitutional neurorights amendment and a 2023 Chilean Supreme Court ruling on neural data protection (Cornejo-Plaza et al., 2024). The RLbF consent framework (Section 8.3) maps to cognitive liberty and mental privacy; the empathic/persuasive boundary (Section 8.1) addresses mental integrity; and the autonomy safeguards (Section 8.4) protect psychological continuity.

The UNESCO Recommendation on the Ethics of Neurotechnology (UNESCO, 2025) provides a third governance reference point. Two provisions are particularly relevant to RLbF. First, the Recommendation's principle of mental privacy – that neural data should be afforded protections at least equivalent to those for health data and that individuals retain the right to control access to information derived from their brain activity – is directly implemented by RLbF's on-device data minimization architecture, which ensures that raw EEG never leaves the acquisition device and that only derived cognitive state scores are transmitted (Section 8.3). Second, the Recommendation's emphasis on cognitive liberty – that neurotechnologies must not be used to manipulate individuals' mental processes without their free and informed consent – maps to the empathic/persuasive boundary formalized in Section 8.1 and the consent requirements that mandate disclosure of all reward function objectives.

<h3>8.1 Empathic vs. Persuasive Computing</h3>

A system that reads cognitive signals and modifies its behavior raises fundamental questions about manipulation and autonomy. We define empathic computing as a paradigm where the system adapts to the user's cognitive state without driving that state toward a predetermined target; the user's state is the input, not the target. Persuasive computing (Fogg, 2003) uses cognitive state knowledge to drive the user toward an external goal (maximizing engagement, inducing purchases). Three testable criteria distinguish the paradigms: the presence of a target state, the direction of optimization (system behavior vs. user state), and the disclosure of objectives.

The default RLbF configuration (<em>w</em><sub>app</sub> = 0) eliminates application-layer goals. However, we must be precise: <em>R</em><sub>res</sub> still encodes normative assumptions about desirable communication dynamics (manageable workload, reduced stress, maintained focus). These internally encoded communication-quality goals are defensible as beneficent defaults but are still goals. More precisely, <em>w</em><sub>app</sub> = 0 precludes externally imposed goals while retaining internally encoded goals about communication quality. This is the difference between "the system helps you process information" and "the system drives you toward a behavior" – a meaningful distinction that should not be overstated. It should also be noted that the default communication-quality targets embedded in <em>R</em><sub>res</sub> – what constitutes manageable workload, appropriate emotional tone, and desirable information density – are culturally and contextually contingent; norms for directness, cognitive challenge, and affective expression vary across cultures, professional domains, and individual preferences, and deployments outside the training population's cultural context may require recalibration of these defaults.

A deeper philosophical complication arises from the prediction model <em>f</em><sub>ϕ</sub>. Criterion 2 claims that an empathic system optimizes the system's behavior rather than the user's state. However, <em>f</em><sub>ϕ</sub> is trained to predict cognitive state transitions: <strong>ŝ</strong><sub>t+1</sub> = <em>f</em><sub>ϕ</sub>(<em>c</em><sub>t</sub>, <strong>s</strong><sub>t</sub>, <em>u</em><sub>t</sub>). In learning this mapping, <em>f</em><sub>ϕ</sub> necessarily acquires a model of the causal chain from utterance properties to cognitive state changes — it learns which utterance features produce which cognitive effects. During RL training, the policy π<sub>θ</sub> uses <em>f</em><sub>ϕ</sub>'s predictions to select utterances that maximize <em>R</em><sub>res</sub>, which is a function of the predicted next cognitive state. The optimization is formally over the policy's action space (word choice, pacing, information density), but the reward gradient flows through a learned model of user state dynamics. The system is, in effect, selecting actions for their predicted effects on the user's cognitive state, even though the optimization variable is the system's output rather than the user's brain. This is analogous to the distinction between a thermostat – which "optimizes its behavior" by adjusting heating output but does so precisely to change room temperature – and a device that targets temperature directly. The empathic/persuasive boundary therefore rests not on whether the system models user state consequences (it necessarily does) but on whether the system pursues an externally imposed target state versus adapting to the user's current state without a predetermined destination. We regard this as an honest complication rather than a fatal objection: the Criterion 2 claim should be understood as "the system adapts its behavior in response to user state" rather than the stronger and less defensible "the system has no model of how its behavior affects user state."

<h3>8.2 Informational Power Asymmetry</h3>

A fundamental informational asymmetry exists – one that raises neurorights concerns (Yuste et al., 2017; Ienca and Andorno, 2017): the model has real-time access to the user's cognitive state while the user has no reciprocal access to the model's strategy. This asymmetry persists even under transparency safeguards, because the model has an integrative advantage (processing scores across time at token-generation speed), a training advantage (learned patterns from thousands of interactions), and a speed advantage (adaptation occurs before the user is consciously aware). Transparency (the user can see their data) is achievable and necessary; full symmetry (equivalent understanding of interaction dynamics) is not achievable. We propose mitigations – real-time adaptation explanations, model-state disclosure, periodic asymmetry audits, and a right to cognitive opacity – while acknowledging that the asymmetry is a permanent structural feature.

Critically, the informational asymmetry is not static: it grows with model capability and accumulated training data. As the model is trained on more interaction sessions, its training advantage deepens since it has observed a wider range of cognitive response patterns and learned subtler correlations between utterance features and cognitive state transitions. As model architectures become more capable (larger context windows, more expressive policy networks), the integrative advantage strengthens, and the model can exploit longer temporal dependencies in cognitive state trajectories that no human interlocutor could track. This scaling dynamic means that the mitigations proposed above are not a one-time implementation but a continuously evolving requirement: the strength of transparency safeguards, the granularity of adaptation explanations, and the frequency of asymmetry audits should scale proportionally with the model's informational advantage. A deployment that was ethically adequate at one capability level may become inadequate after further training or architectural improvement. We therefore recommend that any RLbF deployment include a capability-proportional safeguard policy that triggers reassessment of mitigation adequacy whenever the model undergoes significant retraining or capability expansion.

<h3>8.3 Mental Privacy and Consent</h3>

EEG data is not like behavioral clickstream data. It is an involuntary biological signal that reveals aspects of cognitive processing the user may not be aware of and cannot suppress. The RLbF architecture enforces a critical data minimization choice: raw EEG never leaves the device; only derived cognitive scores are transmitted. This irreversible transformation supports privacy while preserving the information needed for adaptation. It is designed to increase brain capital and the mental wealth of each individual user, not lessening either – thus consent must be specific (naming monitored dimensions), comprehensible, granular (allowing partial consent), revocable with immediate effect, and ongoing.

Critically, informed consent for RLbF systems should be dynamical and updated to include explicit risk disclosure, a requirement absent from many BCI research protocols and one that IRB review processes have not yet standardized for neurodata-based AI. The consent process must disclose, at minimum: (a) that the system learns patterns in the user's cognitive responses and adapts its behavior accordingly, which may subtly shape the user's communication expectations over time; (b) that the informational asymmetry described in Section 8.2 is a permanent structural feature, not a temporary limitation; (c) that long-term cognitive effects of sustained adapted communication are unknown and have not been studied longitudinally; and (d) that cognitive data decodings, while derived from validated models, are probabilistic inferences about internal states and are subject to incidental measurement error, individual variations, and biases in the underlying EEG models. These risk disclosures go beyond the standard IRB-required statement that "participation may involve unknown risks" and treats participants as intelligent, active co-creators with the specific information communicated to them in a way they understand, which is needed to make genuinely informed decisions about their sustained neurodata driven interactions.

Beyond risk disclosure, the consent framework also should address benefits, compensation, and injury provisions. Participants should be informed of the anticipated direct benefits of participation (e.g., exposure to communication adapted to their cognitive state, contribution to research on empathic AI) as well as the realistic possibility that no direct benefit may accrue, particularly in early-stage research where the system's adaptive behavior may be unreliable or imperceptible. Compensation structures for research participants should be designed to avoid undue inducement while fairly valuing the time, cognitive effort, and neural data contributed; this is especially important given that EEG-based participation is more invasive and time-intensive than standard survey-based AI research.

Finally, the consent process must specify provisions for participant injury, including both physical harm (e.g., discomfort from prolonged EEG device wear, skin irritation from electrodes) and psychological harm (e.g., distress caused by the system's maladaptive responses to misinterpreted cognitive states, or anxiety arising from awareness of continuous cognitive monitoring). Participants must be informed of available recourse, including procedures for reporting adverse experiences, access to appropriate clinical support, and the circumstances under which the research team will provide or facilitate medical or psychological care.

In multi-party settings, all participants (including non-monitored ones) must be informed that the model adapts to monitored users' cognitive states, with transparent policies for priority resolution and opt-out mechanisms for non-monitored participants. An additional consent requirement arises when session recordings are used as training data (Phases 2 and 3): because multi-party interaction logs encode the communicative behavior and contextual contributions of all participants, consent for use in model training must be obtained from every participant in the session, not only those whose EEG was recorded.

Under the EU General Data Protection Regulation (GDPR), EEG-derived data such as cognitive workload, enjoyment, and focus constitute special category data within the meaning of Article 9, because they are inferred from biological signals and reveal information about an individual's mental health and cognitive processing. Any RLbF deployment processing such data in jurisdictions subject to the GDPR must therefore obtain explicit consent under Article 9(2)(a), conduct a Data Protection Impact Assessment (DPIA) as required by Article 35 for high-risk profiling of sensitive data, and honor the data portability right under Article 20 by enabling users to export their cognitive state logs in a structured, machine-readable format.

Article 22 of the GDPR, which grants individuals the right not to be subject to decisions based solely on automated processing that produce legal or similarly significant effects, requires specific analysis in the RLbF context. An RLbF system that autonomously adapts its communication based on real-time cognitive state inference is engaged in automated decision-making about how to interact with the user – decisions that, in certain deployment contexts, could produce significant effects on the individual. In therapeutic settings, automated adaptation decisions could influence treatment outcomes; in educational contexts, they could affect learning trajectories and assessments; in employment-related communication, they could shape professional evaluations. While the adaptive communication decisions in a general conversational context may not rise to the level of "legal or similarly significant effects," deployers must conduct a context-specific Art. 22 analysis for each use case. Where Art. 22 applies, the deployment must implement meaningful human oversight (not merely nominal human-in-the-loop), provide the user with information about the logic involved in the automated processing, and offer mechanisms to contest the automated decision and obtain human review. The RLbF framework's existing transparency safeguards (adaptation explanations, model-state disclosure) provide a foundation for Art. 22 compliance, but they must be supplemented with explicit contestation mechanisms and genuine human oversight authority in high-stakes deployment contexts.

In the United States, the Health Insurance Portability and Accountability Act (HIPAA) imposes specific obligations when RLbF is deployed in therapeutic or clinical contexts. Where an RLbF system is integrated into a healthcare provider's services (for example, as a communication aid in teletherapy, cognitive rehabilitation, or mental health support) the cognitive state scores derived from EEG constitute protected health information (PHI) under HIPAA's Privacy Rule (45 CFR Part 160 and Subparts A and E of Part 164). This classification triggers requirements including: designation of the RLbF platform operator as a business associate with an executed Business Associate Agreement (BAA); implementation of administrative, physical, and technical safeguards under the Security Rule (45 CFR Part 164, Subparts A and C); minimum necessary use limitations restricting cognitive state data access to the specific dimensions required for the therapeutic purpose; individual rights to access and amend their cognitive state records; and breach notification obligations within 60 days of discovery for unauthorized disclosure of cognitive state data. The on-device processing and score-only transmission architecture described above aligns well with HIPAA's minimum necessary standard, but deployers must additionally ensure that cognitive state logs stored for longitudinal monitoring (Section 8.5) are encrypted at rest and in transit, that access controls restrict data to authorized clinical personnel, and that de-identification procedures meet HIPAA's Safe Harbor or Expert Determination standards before any use in research or model training. Analogous obligations apply under other health-data regulations (e.g., the Personal Information Protection Law (PIPL) in China) and emerging neurotechnology governance frameworks (UNESCO, 2025; Cornejo-Plaza et al., 2024). Deployers should map the consent framework described above to the applicable regulatory regime before any user-facing trial.

In addition to essential data protections, the EU Artificial Intelligence Act (Regulation 2024/1689) introduces a risk-based classification framework with direct implications for RLbF deployment. An RLbF system is likely to be classified as high-risk under Annex III of the AI Act on at least two grounds. First, Category 1(a) covers AI systems intended to be used as safety components of, or that themselves constitute, biometric identification or categorization systems, and EEG-derived cognitive state inference may qualify as biometric categorization of natural persons based on physiological signals. Second, if RLbF is deployed in educational or vocational training contexts (Category 3) or in contexts that affect access to essential services (Category 4), the system falls squarely within the high-risk classification. High-risk classification triggers obligations under Title III, Chapter 2 of the AI Act: a risk management system maintained throughout the AI system's lifecycle (Art. 9), data governance requirements including examination for possible biases in the training data (Art. 10), technical documentation sufficient for conformity assessment (Art. 11), record-keeping and automatic logging of system operation (Art. 12), transparency obligations including user-facing information about the system's capabilities and limitations (Art. 13), human oversight provisions enabling human intervention and override (Art. 14), and accuracy, robustness, and cybersecurity requirements (Art. 15). Several of these requirements align with safeguards already proposed in the RLbF framework: the ethical review checklist (Section 8.4) addresses transparency and human oversight, and the data minimization architecture satisfies the spirit of data governance. However, conformity assessment, lifecycle risk management, and the specific technical documentation requirements of Art. 11 would require dedicated compliance infrastructure beyond what the current framework specifies. Deployers targeting EU markets should conduct a formal AI Act classification analysis and engage with the relevant national supervisory authority before any user-facing deployment.

<h3>8.4 Safeguards and Oversight</h3>

Architectural safeguards include user control (hard off-switch for adaptation, granular dimension controls, explicit preference overrides), no hidden objectives (disclosure and separate consent for any <em>w</em><sub>app</sub> &gt; 0 deployment), and anti-sycophancy guardrails (diversity constraints, challenge injection, periodic baseline exposure). The risk that sustained adaptation may erode users' capacity to process unadapted communication is a specific instance of cognitive offloading (Risko and Gilbert, 2016): when an external system reliably performs a cognitive function, individuals reduce their own investment in that function. Periodic baseline exposure is designed to counteract this tendency by maintaining the user's independent communicative resilience.

We recommend that baseline exposure sessions – in which the system disables cognitive state adaptation and operates with standard, non-adapted output – occur at a minimum frequency of one session per five adapted sessions (or at least once per week for daily users), with each baseline session lasting at least 15 minutes of continuous interaction. Users must be permitted to opt out of baseline exposure; however, opting out should trigger a logged advisory, and sustained opt-out (more than four consecutive weeks) should be flagged for review by the deployment oversight body. The ethical review checklist for RLbF deployments covers:

<ol>
<li>Reward function transparency – all reward components and weights are documented and accessible to auditors. <strong>Blocking</strong>: deployment must not proceed without published reward specification.</li>
<li>Target state verification – formal confirmation that no externally imposed target cognitive state is encoded in the reward function (or, if <em>w</em><sub>app</sub> &gt; 0, that the target is disclosed). <strong>Blocking</strong>.</li>
<li>Data minimization – only derived cognitive scores (not raw EEG) leave the acquisition device. <strong>Blocking</strong>.</li>
<li>User control – hard off-switch, granular dimension controls, and explicit preference overrides are functional and tested. <strong>Blocking</strong>.</li>
<li>Consent adequacy – consent satisfies specificity, comprehensibility, granularity, revocability, and risk disclosure requirements (Section 8.3). <strong>Blocking</strong>.</li>
<li>Anti-sycophancy measures – diversity constraints, challenge injection, and baseline exposure protocol are active and parameterized. <strong>Advisory</strong>: absence does not prevent deployment but triggers enhanced monitoring.</li>
<li>Autonomy safeguards – baseline exposure schedule is implemented and opt-out logging is operational. <strong>Advisory</strong>.</li>
<li>Human oversight for high-stakes contexts – a designated human reviewer is assigned for therapeutic, educational, or crisis-adjacent deployments. <strong>Blocking</strong> for high-stakes contexts; <strong>advisory</strong> otherwise.</li>
<li>Asymmetry mitigations – real-time adaptation explanations and periodic asymmetry audits are scheduled. <strong>Advisory</strong>.</li>
<li>Third-party consent – all non-monitored participants in multi-party settings have been informed and have opted in. <strong>Blocking</strong> for multi-party deployments.</li>
<li>Implicit goal disclosure – internal communication-quality objectives encoded in <em>R</em><sub>res</sub> are documented and disclosed to users. <strong>Advisory</strong>.</li>
<li>Incident response mechanism – a documented protocol exists for responding to adverse cognitive events (e.g., acute distress, unanticipated cognitive state patterns) detected during deployment, including escalation criteria and responsible parties. <strong>Blocking</strong>.</li>
<li>Version control and re-consent – significant model updates (retraining, reward function modification, or architecture changes) trigger re-consent from active users and version-stamped audit logs linking each user session to the model version in use. <strong>Blocking</strong>.</li>
<li>Acknowledgement that EEG is intrinsically Personally Identifiable Information (PII) data and that data handling must accordingly abide by all local, state and federal laws of the United States of America pertaining to Personally Identifiable Information. <strong>Blocking</strong>.</li>
</ol>

The strongest ethical case for RLbF is structural: <em>w</em><sub>app</sub> = 0 removes the mechanism for externally specified manipulative objectives. The strongest concerns are that implicit goals can produce harmful emergent behaviors (sycophancy, echo chambers, cognitive narrowing) and that informational asymmetry creates structural conditions for exploitation. The safeguards make risks visible and manageable, not absent.

<strong>Long-term cognitive effects.</strong> The attunement paradox described above – the risk that effective adaptation reduces the user's own adaptive capacities – is an instance of the broader cognitive offloading phenomenon studied in cognitive psychology (Risko and Gilbert, 2016). When individuals routinely delegate cognitive tasks to external tools, the corresponding cognitive skills atrophy: GPS navigation erodes spatial reasoning, calculators diminish mental arithmetic, and spell-checkers weaken orthographic memory. RLbF's communication adaptation constitutes cognitive offloading of the skill of parsing imperfect communication – a skill that is exercised in virtually every human interaction and whose erosion would have pervasive consequences. Three specific long-term effects warrant attention.

First, <em>skill atrophy</em>: sustained exposure to communication optimized for the user's cognitive state may degrade the user's capacity to extract meaning from communication that is poorly structured, ambiguously worded, emotionally mismatched, or delivered at an inappropriate level of complexity. This capacity is a foundational social skill, and its degradation would impair the user's ability to function in unadapted human communication environments.

Second, <em>communication expectation calibration</em>: users habituated to cognitively attuned communication may develop expectations that human interlocutors cannot meet, leading to frustration, disengagement, or social withdrawal when interacting with non-adaptive conversation partners. This effect is analogous to the documented phenomenon of users developing unrealistic expectations of conversational quality after sustained interaction with AI assistants.

Third but not least, <em>metacognitive effects</em>: if the system's adaptation is sufficiently seamless, users may lose awareness of their own cognitive state dynamics, including the effort of comprehension, the experience of confusion, the satisfaction of working through a difficult idea, because these experiences are managed away before conscious registration. This metacognitive flattening could impoverish the user's self-understanding and reduce their capacity for self-directed cognitive regulation. The periodic baseline exposure safeguard partially addresses these risks, but its efficacy depends on implementation specifics that remain unresolved: the frequency and duration of baseline episodes, whether users can opt out (which would render the safeguard toothless), and whether brief periodic exposure is sufficient to prevent skill atrophy from sustained adapted interaction between exposures. Longitudinal studies measuring communication competence, interpersonal satisfaction in unadapted settings, and metacognitive awareness across sustained RLbF use are essential before any deployment beyond research contexts.

<h3>8.5 Vulnerable Populations</h3>

The ethical framework described above treats users as a homogeneous category, but certain populations face heightened risks from cognitive-state-adaptive AI systems and require population-specific safeguards.

<strong>Children and adolescents.</strong> Developing brains exhibit different EEG signatures than adult brains, and cognitive state models trained on adult populations may produce systematically inaccurate inferences when applied to minors (Saby and Marshall, 2012). More fundamentally, children and adolescents are in a formative period during which communication skills, cognitive resilience, and metacognitive capacities are actively developing. A system that adapts communication to minimize cognitive difficulty could interfere with the developmental processes through which young users learn to parse complex language, tolerate ambiguity, and engage with cognitively demanding material. Any RLbF deployment involving minors should require (a) age-appropriate recalibration of cognitive state models using pediatric EEG norms, (b) parental or guardian consent in addition to age-appropriate assent, (c) heightened baseline exposure requirements (Section 8.4) to protect developmental acquisition of communication skills, and (d) exclusion from any deployment with <em>w</em><sub>app</sub> &gt; 0 pending dedicated ethical review.

<strong>Individuals with mental health conditions.</strong> Users experiencing depression, anxiety disorders, PTSD, or other mental health conditions present both the strongest use case and the greatest risk for empathic AI. On one hand, these users may benefit most from communication that is calibrated to their cognitive state: reduced information density during high-stress episodes, for example. On the other hand, their cognitive state signals may be atypical, their vulnerability to dependence on adapted communication may be elevated, and the boundary between empathic support and therapeutic intervention becomes blurred. RLbF systems must not be represented as therapeutic tools unless they have undergone clinical validation and regulatory review as medical devices. Deployments in contexts where users with mental health conditions are likely to be present, such as counseling platforms, crisis services, health information systems, should implement additional safeguards: clinical oversight by a licensed mental health professional, real-time monitoring for cognitive state patterns indicative of acute distress, and clear escalation protocols that route users to human care when the system detects states outside its competence boundary.

<strong>Neurodivergent users.</strong> Individuals with autism spectrum conditions, ADHD, dyslexia, or other neurodevelopmental differences may exhibit cognitive state patterns that diverge systematically from the population norms on which <em>f</em><sub>ϕ</sub> is trained. A model calibrated to neurotypical EEG patterns may misinterpret neurodivergent cognitive states: for instance, interpreting atypical attention patterns as disengagement when they reflect a different but functional mode of cognitive processing. This is not merely a measurement error problem but a normative one: the "desirable" cognitive state trajectory encoded in <em>R</em><sub>res</sub> reflects neurotypical assumptions about what constitutes manageable workload, appropriate focus, and reduced stress. Neurodivergent users may thrive under cognitive conditions that the system treats as suboptimal. Addressing this requires (a) explicit inclusion of neurodivergent users in training populations with distinct user embeddings, (b) user-configurable cognitive state targets that override population-level defaults, and (c) transparency about the normative assumptions embedded in the default reward function.

<strong>Elderly users and users with cognitive decline.</strong> Older adults, particularly those with early-stage cognitive decline or dementia, face a distinctive version of the attunement paradox: a communication-adaptive system that compensates for declining cognitive capacity may mask early symptoms of deterioration that would otherwise prompt clinical evaluation. If the system continuously simplifies its language in response to declining comprehension scores, neither the user nor their caregivers may recognize the progression of impairment. Safeguards for elderly populations should include (a) longitudinal monitoring of cognitive state baselines to detect secular trends that may indicate clinical relevance, (b) caregiver notification protocols (with user consent) when sustained baseline shifts exceed predefined thresholds, and (c) periodic clinical review requirements for long-term deployments in elder care settings.

<h3>8.6 Equity and Access</h3>

Yuste et al.'s (2017) fourth ethical priority – equality – warns that cognitive enhancement technologies risk exacerbating existing social inequalities. RLbF confronts this concern at multiple levels, and the current framework does not resolve it through architectural design alone.

<strong>The digital divide.</strong> RLbF training requires participants who own consumer-grade EEG devices, which currently cost several hundred dollars and are marketed primarily to technology enthusiasts in high-income countries. The training population is therefore likely to be biased toward young, educated, affluent individuals from Western, educated, industrialized, rich, and democratic (WEIRD) societies (Henrich et al., 2010). Cognitive response patterns learned from this population may not generalize to users from different socioeconomic, educational, or cultural backgrounds. Even if the trained model is deployed without EEG hardware (via open-loop transfer), its learned communication policy reflects the cognitive preferences and response patterns of the training population. Users from underrepresented backgrounds may receive communication that is optimized for cognitive dynamics they do not share. This is not a hypothetical concern: it is structurally identical to the well-documented problem of training data bias in machine learning systems more broadly, but with the added dimension that the bias operates on neurophysiological response patterns rather than behavioral preferences. Addressing this requires deliberate recruitment of diverse training populations across socioeconomic strata, geographic regions, educational backgrounds, and age groups, as well as evaluation of the trained model's communication effectiveness across demographic categories not represented in training.

<strong>Cultural bias in cognitive state norms.</strong> The cognitive state constructs operationalized in the RLbF reward function – workload, enjoyment, focus – are not culturally neutral. What constitutes "manageable workload" varies across cultures with different expectations for cognitive effort in communication. In high-context communication cultures (Hall, 1976), a substantial proportion of meaning is conveyed through implication, shared context, and indirect reference, and the cognitive work of interpretation is valued rather than minimized. A model trained to reduce cognitive load may produce communication that is perceived as overly explicit, condescending, or culturally inappropriate in high-context settings. Similarly, norms for emotional expressiveness, appropriate levels of directness, and tolerance for ambiguity vary across cultures in ways that a reward function calibrated to one cultural context may not accommodate. The user-configurable cognitive state targets proposed in Section 8.5 offer a partial remedy, but they place the burden of cultural accommodation on the individual user rather than incorporating cultural variation into the system's default behavior. A more robust approach would include culturally differentiated baseline models and explicit disclosure of the cultural assumptions embedded in the default reward parameterization.

<h3>8.7 Dual-Use Scenario Analysis</h3>

The dual-use risk of RLbF extends beyond the general observation that empathic technology can be repurposed for manipulation. Three specific misuse scenarios warrant analysis because they illustrate distinct mechanisms through which the framework's capabilities could be weaponized, and because each requires different countermeasures.

<strong>Coercive interrogation and deception detection.</strong> An RLbF system deployed with <em>w</em><sub>app</sub> &gt; 0 could be configured to optimize for information extraction: the application-layer reward could incentivize the model to produce utterances that maximize the subject's cognitive load (to induce stress and reduce executive function), that exploit detected moments of low focus (to insert leading questions when the subject's guard is down), or that calibrate emotional tone to build false rapport. In an interrogation context, the real-time cognitive state feedback provides the interrogator – or the automated system – with a continuous lie-detection-adjacent signal that reveals which questions produce elevated stress responses, which topics trigger cognitive avoidance, and when the subject's mental defenses are depleted. Even without explicit deception detection, the system's ability to adaptively adjust conversational pressure based on real-time cognitive state monitoring constitutes a qualitative escalation in interrogation capability. Countermeasures include regulatory prohibition of RLbF deployment in custodial, law enforcement, or military interrogation contexts, and technical restrictions that prevent reward function configurations from optimizing for cognitive state patterns associated with distress or coercion.

<strong>Addictive engagement optimization.</strong> A commercial deployer could configure <em>R</em><sub>app</sub> to maximize session duration, return frequency, or emotional dependency – standard engagement metrics in the attention economy. The RLbF architecture makes this unusually effective because the system can detect, in real time, which conversational strategies trigger the strongest positive cognitive responses (enjoyment spikes, flow states) and which create the aversive states (boredom, frustration) that precede session termination. Combined with the speed advantage described in Section 8.2 (adaptation occurs before conscious awareness) this creates a mechanism for producing compulsive engagement patterns that bypass the user's deliberative assessment of whether continued interaction is in their interest. The structural analogy to variable-ratio reinforcement schedules in gambling (Schull, 2012) is direct: the system can learn to vary its reward delivery (moments of high communicative resonance interspersed with controlled frustration) to maximize behavioral persistence. Countermeasures include prohibiting engagement-metric-based reward components in <em>R</em><sub>app</sub>, mandating session duration limits with mandatory cooling-off periods, and requiring independent audit of deployed reward functions to verify that no engagement-maximizing objective is operative.

<strong>Political influence and opinion manipulation.</strong> An RLbF system trained to monitor cognitive responses to politically relevant content could identify which framing strategies, emotional appeals, and information-density levels are most effective at shifting a specific user's cognitive engagement with political topics. Unlike current targeted political advertising, which infers political susceptibility from behavioral traces (clicks, likes, browsing history), an RLbF system would have access to real-time neurophysiological responses, enabling it to detect cognitive resistance to a message, identify the precise moment when a user's skepticism diminishes, and adaptively calibrate persuasive messaging to each individual's cognitive profile. This capability could be deployed at scale for political campaigns, state propaganda, or radicalization. Countermeasures must operate at the regulatory level (extending political advertising transparency requirements to cognitive-state-adaptive AI systems and prohibiting the use of neurophysiological data for political targeting) and at the technical level (restricting access to the cognitive state API for applications that involve political, ideological, or religious content, and implementing domain-specific deployment restrictions in the platform's access control layer).

These scenarios are not exhaustive, but they illustrate that the dual-use risk of RLbF is not simply that "empathic technology could be misused": it is that the specific capabilities of real-time cognitive state feedback create qualitatively new attack surfaces that existing regulatory frameworks were not designed to address.

<h2>9. Discussion</h2>

<h3>9.1 Implications for the LLM Post-Training Community</h3>

RLbF introduces a fundamentally new class of reward signal for language model post-training, replacing the discrete, voluntary feedback of RLHF (Section 1.2) with involuntary, continuous, real-time measurements of cognitive impact. This distinction has consequences beyond communication style.

The temporal density of the RLbF signal could enable finer-grained credit assignment than any existing post-training method, potentially attributing cognitive state changes to specific sentences or sub-sentence transitions. This temporal resolution could inform better reward modeling even within the standard RLHF paradigm: for instance, by identifying which parts of a response drive positive or negative evaluations.

The involuntary nature of the signal offers a partial deconfounding of "preferred" from "genuinely beneficial." RLHF-induced sycophancy (Sharma et al., 2024) arises precisely because voluntary feedback rewards matching stated preferences. If RLbF's brain feedback correlates with genuine communicative benefit, it could produce less sycophantic models – a hypothesis that motivates the diagnostic tests in Section 6.

More broadly, RLbF suggests that the post-training community should consider feedback signals beyond human judgments and AI evaluations. Physiological signals (not just EEG but potentially eye tracking, skin conductance, or other biosignals) represent an underexplored category of reward signals that could complement existing methods. As John Searle put it in "Biological Naturalism", 2004: "The fact that brain processes cause consciousness does not imply that only brains can be conscious. The brain is a biological machine. Because we do not know exactly how the brain does it we are not yet in a position to know how to do it artificially."

<h3>9.2 Implications for the BCI and Neurotechnology Community</h3>

RLbF proposes a novel BCI application (Wolpaw et al., 2002): not communication or control, but reward signal generation for AI training. This inverts the typical BCI paradigm, where neural signals are the output (decoded into commands or communications) rather than the feedback (shaping another system's behavior). Within Zander and Kothe's (2011) taxonomy, this places RLbF in the passive BCI category – the user performs no deliberate mental task; their cognitive state is monitored as they naturally engage with generated text.

The closed-loop architecture demonstrates a new modality for BCI interaction. Traditional passive BCIs monitor cognitive state for logging or simple parameter adjustment. RLbF's closed loop is tighter: cognitive state directly and continuously shapes the content of a generative language model's output. This creates a form of human-AI interaction where the user's brain is, in a meaningful sense, co-producing the conversation, not through deliberate commands but through the involuntary cognitive signals that guide the model's adaptive communication.

For the neurotechnology community, RLbF provides a concrete downstream application that motivates improvements in cognitive state decoding. The framework's utility is bounded by decoder accuracy: if decoded scores do not reliably reflect cognitive constructs, the reward signal is noise. This creates a clear value proposition for decoder development: better cognitive state decoding is expected to translate to better-trained empathic models. The minimum decoder accuracy requirements for useful RLbF training remain an empirical question, but the framework provides a principled benchmark for evaluating decoder improvements in terms of their downstream effect on language model communication quality.

A second implication is data production. Real-world RLbF deployments could produce conversation-coupled EEG datasets that are rare in today's literature: utterance histories, response timing, EEG traces, decoder confidences, and downstream outcomes collected in the same interactive loop. In principle, high-confidence decoder outputs could supervise some segments directly, while ambiguous segments might benefit from context-based weak supervision from language models or semi-supervised correction methods. We emphasize that such labels would be pseudo-labels, not ground truth, and would require strict calibration, agreement filters, and held-out validation to avoid circularity.

More broadly, RLbF suggests that the neurotechnology and language model communities may benefit from collaboration at the optimization level (using neurophysiological signals to guide language model behavior) alongside work at the representation level. This optimization agenda is adjacent to, but distinct from, most current EEG foundation model research, and it may be especially actionable in the near term given present limits in transfer and interactive data coverage.

<h3>9.3 Current Limitations and Open Challenges</h3>

We identify the following limitations of the current work:

<strong>Theoretical, not proven yet by empirical results from experiments.</strong> The most fundamental limitation is that every empirical claim in this paper is a hypothesis still in need of testing quantitatively in real world conditions. The framework is theoretical and this arXiv version is intended to establish the formalization, the architecture, and the experimental design in a clear manner that will invite in additional researchers to collaborate since these questions are complex and multifaceted for any single group to work on. The goal is for a subsequent journal version to report empirical evidence and conclusions about the theory's utility, accompanied by a shared repository of RLbF datasets to support replication and extension by the wider research community.

<strong>The Signal-Processing Bottleneck: Distributional Shift and Deconvolution.</strong> Capturing the distributional shift and deconvolution of users' cognitive responses remains an open, and complex challenge — presenting a fundamental identifiability problem (cf. Pearl, 2009) and a non-trivial transfer learning problem overlapped regarding within-session domain and distributional shifts. Our proposed linear subtraction model serves merely as a first-order approximation for this problem. It is almost certainly an over-simplification, relying on a 60-second window assumes that the user's cognitive state within that timeframe averages out to a neutral baseline which we viscerally understand as humans to not be the natural truth.

This assumption breaks down especially during periods of sustained cognitive effort; if a user maintains a state of high workload for a full minute, the moving average will artificially depress the subsequent response estimates. Second, correcting drift via subtraction implies that the shift is purely additive and confined to the latent prediction space. In reality, physiological drift often involves complex transformations—such as spatial rotations or non-linear changes in the raw EEG space—which require advanced transfer learning techniques to adequately correct. Finally, capturing highly ephemeral, short and long latency evoked potentials and event-related potential trains triggered by specific lexical items requires more sophisticated deconvolution models to disentangle the neurological reactions from the broader, second-level averaged cognitive state.

<strong>Hardware dependency and population bias.</strong> RLbF training requires EEG-equipped participants, limiting the training population to users who own EEG devices and consent to wear them. This population is likely biased toward young, educated, technology-enthusiastic individuals from wealthy countries. In other words, statistically WEIRD people when compared to the entire global population (from Western, Educated, Industrialized, Rich, and Democratic societies). Cognitive response patterns learned during training thus may not generalize to billions of people across all swaths of human society, that remains an empirical question.

<strong>Decoder dependency.</strong> The framework's utility depends on the accuracy of the cognitive state decoder. If decoded scores are noisy or biased, the reward signal is correspondingly degraded. What "sufficient accuracy" means for RLbF training convergence is itself an empirical question.

<strong>Weak-supervision circularity.</strong> If future RLbF systems use context models or LLMs to propose labels for uncertain EEG segments, those pseudo-labels could amplify model biases or create self-confirming training loops. Any such pipeline would need uncertainty calibration, conservative promotion rules, and held-out human-reviewed validation sets before pseudo-labels are treated as training targets.

<strong>Resonance-reward operationalizations.</strong> The specific target functions for information density matching and emotional tone alignment are starting points, not optimized solutions. They depend on the choice of reference language model and sentiment classifier, creating an operational under-determination that different implementations could resolve differently, producing different trained behaviors.

<strong>Voice-mode limitations.</strong> The temporal model handles text-based turn-by-turn interaction well but provides only paragraph-level reward attribution for streaming voice delivery, where overlapping utterances create entangled cognitive responses.

<strong>Measurement reactivity.</strong> Wearing an EEG device during conversation is not ecologically neutral. The awareness of being monitored may alter cognitive responses (a form of Hawthorne effect), and training data collected under monitoring conditions may not generalize perfectly to unmonitored interaction. The open-loop transfer design partially addresses this – the model operates without monitoring at inference – but the training data itself is collected under monitoring.

<strong>Ethical irreducibility.</strong> The informational power asymmetry between a cognitive-state-aware model and a cognitively opaque user is a permanent structural feature that cannot be fully eliminated by transparency safeguards, especially in real-time. The proposed mitigations reduce harm potential but do not equalize the fundamental imbalance. Similarly, the internally encoded communication-quality goals in <em>R</em><sub>res</sub> (even with <em>w</em><sub>app</sub> = 0) constitute a normative stance about desirable cognitive states that users may not share in all contexts.

<h2>10. Conclusion</h2>

This paper introduces RLbF as a framework for training language models with decoded neurophysiological feedback. The core idea is to separate EEG decoding from language-model alignment: a decoder estimates cognitive state from EEG, and those estimates are used as a noisy continuous reward for post-training. This framing leaves room for multiple decoder families, including future EEG foundation models, while focusing this paper's main contribution on the alignment layer – presenting a formal specification of a cognitive state space, the POMDP interaction loop, a temporal delay model, and a three-component reward function with concrete, testable operationalizations grounded in Cognitive Load Theory and therapeutic alliance research.

The three-phase training pipeline is designed to transform a standard instruction-tuned LLM into an empathic agent through progressive stages: learning the vocabulary of cognitive state from synthetic data, grounding it in real human cognitive responses from Isaac session recordings, and optimizing communication policy against the full empathic reward via PPO. The system architecture, realized in the Isaac platform, implements the closed-loop cognitive feedback loop and is designed to evolve from a dual-agent stepping stone to a single post-trained empathic model. We also advance a data-generation hypothesis: if deployed responsibly, closed-loop conversational systems may generate ecologically valid datasets that pair EEG activity with conversation context, model behavior, decoder confidence, and downstream outcomes. Such datasets could later support better cognitive-state predictors and perhaps future EEG foundation models specialized for interactive settings, although this possibility remains prospective and unvalidated in the present work.

This paper's boldest claim – the hypothesis that communication skills learned from brain feedback persist when EEG hardware is removed, which we call open-loop empathic transfer – determines whether RLbF scales beyond the EEG-equipped training population to serve any user anywhere, or whether its benefits will remain confined to closed-loop deployments in relatively expensive technological environments. Future research efforts should work to elaborate these relationships more clearly. A step in this direction is our design for a controlled experiment with pre-registered analyses to test it, where if the hypothesis holds, RLbF will show to offer a path to language models that communicate not just informatively and helpfully, but with genuine attunement to their listener's cognitive experience. If it fails, the framework may hold value for instrumented settings but would never converge on the broader ambition of universally empathic communication with systems that are trained to do no harm.

Our mental privacy-preserving ethical framework is not an appendix or a bolt-on feature of this architecture; it is a load-bearing component of the design. The distinction between empathic and persuasive computing, the analysis of informational power asymmetry, the consent requirements for neurological data, and the safeguards against addition and cognitive dependence are as fundamental to the framework's viability as any mathematical formula. We have attempted to be transparent about what we believe the framework can and cannot guarantee: <em>w</em><sub>app</sub> = 0 precludes externally imposed goals but retains internally encoded communication-quality objectives; transparency is achievable but full informational symmetry is not; safeguards make risks visible but do not eliminate them.

Every empirical claim in this paper is either supported by existing evidence or explicitly labeled as a hypothesis pending validation, and this honesty is deliberate. The RLbF framework proposes a fundamentally new relationship between neural signals and language models: one where the goal is not to teach machines to decode brain signals, but to teach machines to communicate in ways that the brain finds genuinely useful. Validating this framework requires data, experiments, and results that are beyond the scope of this paper, which aims to establish a new formal foundation: a precise, testable, ethically grounded framework for language model improvement that includes not just what users say, but how users think and feel.

<h2>Declaration of Interests</h2>

All authors work for Arctop Inc. Some systems described rely on proprietary software platforms developed by Arctop (Furman et al., 2021, US Patent Application US20210390366A1), including empathic computing systems and methods for improved human interactions with digital content experiences. The RLbF framework described in this paper builds on cognitive state decoding technology developed by Arctop and proposes the Isaac platform – which integrates Arctop's real-time EEG decoding – as the implementation architecture (Section 4). This relationship creates a potential conflict of interest in two respects.

First, this paper's assessment of the feasibility and accuracy of real-time cognitive state decoding (Section 3) relies in part on Arctop's published technical capabilities, and the author has a financial interest in the commercial success of that technology. Second, the proposed research program, if successful, could increase the value and adoption of Arctop's decoding platform, creating a direct commercial incentive for the research direction advocated here.

We have attempted to mitigate these conflicts by (a) grounding all empirical claims in published, independently verifiable evidence and clearly labeling all unvalidated claims as hypotheses, (b) designing the RLbF framework to be decoder-agnostic, such that any cognitive state decoder that provides the required score vector could theoretically substitute for Arctop's system, so the framework's value does not depend on a specific commercial use or implementation, and (c) presenting an honest ethical analysis (Section 8) that identifies risks and limitations of the proposed approach without minimizing them for commercial advantage. Readers should nonetheless be aware of this relationship when evaluating this paper's claims about the quality and feasibility of real-time EEG-derived cognitive state decoding in natural environments.

<aside class="dd-footnotes">
<h3>Footnotes</h3>
<ol>
<li id="fn1">Neuro-GPT (Cui et al., 2024) is a separate EEG foundation model evaluated by EEG-Bench but not among the eight models in our systematic review.</li>
<li id="fn2">This also avoids a notation collision with the standard POMDP discount factor γ.</li>
</ol>
</aside>

<h2>Appendix A: EEG Foundation Model Summaries</h2>

This appendix provides per-model summaries for the eight EEG foundation models reviewed in Section 2.4. For the taxonomy, comparative analysis, and key findings, see Sections 2.4.1–2.4.7.

<h3>A.1 BENDr (Kostas et al., 2021)</h3>

The pioneering EEG foundation model, adapting wav2vec 2.0 from speech recognition. A six-layer CNN encoder with 96× downsampling feeds into an 8-layer transformer with contrastive pretraining on TUEG data from over 10,000 subjects. BENDr established the conceptual framework for EEG foundation models. Its limitations — single pretraining dataset, aggressive downsampling discarding high-frequency information, fixed channel assumptions — define the gaps later models address. By 2025, BENDr has been surpassed on all benchmarks but remains the most independently assessed model (appearing in all three benchmark studies). Notably, EEG-Bench found BENDr outperforms LaBraM on epilepsy detection (0.740 vs. 0.565) despite LaBraM's general superiority — a task-specific inversion attributed to label imbalance sensitivity.

<h3>A.2 BIOT (Yang et al., 2023)</h3>

Addresses biosignal heterogeneity through a unified tokenization scheme converting EEG, ECG, and accelerometer data into standardized "biosignal sentences." Linear attention maintains O(n) complexity. At approximately 3.3M parameters, BIOT is the smallest model; its innovation lies in practical cross-modality tokenization rather than absolute performance. Benchmarks show BIOT benefits dramatically from multi-task learning (63.98% to 71.37% balanced accuracy on the workload task in EEG-FM-Bench), suggesting weak pretrained representations can be effectively regularized.

<h3>A.3 LaBraM (Jiang et al., 2024a)</h3>

The first EEG foundation model to demonstrate clear scaling behavior, with three sizes (Base 5.8M, Large 46M, Huge 369M) pretrained on approximately 2,500 hours from roughly 20 diverse datasets. Key innovation: a two-stage approach training a VQ-VAE neural tokenizer (8,192 discrete embeddings using Fourier spectrum targets) followed by masked prediction of these codes. LaBraM-Huge achieves the strongest independently verified results on TUAB (0.826 balanced accuracy) and TUEV (0.662 balanced accuracy). Data scaling analysis found performance plateauing after approximately 1,000 hours of pretraining, challenging the assumption that more data always helps. Serves as the de facto standard comparison point.

<h3>A.4 EEGPT (Wang et al., 2024)</h3>

Addresses the fundamental problem that raw EEG has low signal-to-noise ratio by aligning predictions with high-SNR reference representations rather than reconstructing raw signals. At approximately 10M parameters, achieves performance competitive with LaBraM on several tasks while being far more compact. Notable for claiming strong linear probing results, contradicting the consensus frozen-backbone collapse — a discrepancy likely attributable to differences in evaluation protocol rather than genuinely superior representations.

<h3>A.5 CBraMod (Wang, J. et al., 2025)</h3>

Introduces the criss-cross transformer, separating spatial attention (across channels) and temporal attention (within channels), reducing computational cost by approximately 32% versus full attention. With only approximately 5M parameters pretrained on 27,062 hours of TUEG data, achieves performance comparable to or exceeding models 5–14× its size across the broadest evaluation of any model: 10 downstream tasks on 12 datasets. EEG-FM-Bench identified CBraMod as the standout performer. The most reproducible and practically accessible model, pretrained on a single publicly available dataset. From the same Zhejiang University group as EEGMamba, creating the only quasi-controlled transformer-vs-SSM comparison in the field.

<h3>A.6 EEGMamba (Gui et al., 2025)</h3>

The first EEG foundation model based on a state-space architecture (bidirectional Mamba with Spatio-Temporal-Adaptive module and Task-aware Mixture of Experts). Linear O(n) complexity addresses the quadratic scaling bottleneck for long clinical recordings that routinely span hours, far exceeding the approximately 90-second context limit of transformer-based models. Pretrained on 16,724 hours using masked reconstruction. Critical limitation: zero independent benchmark evaluation.

<h3>A.7 NeuroLM (Jiang et al., 2024b)</h3>

The most radical departure: converting EEG signals into discrete tokens processed by a pretrained GPT-2 backbone. Three-stage pipeline: text-aligned neural tokenizer, autoregressive next-token prediction, multi-task instruction tuning via natural language prompts. Sizes: 254M to 1.7B parameters. The only model enabling multi-task flexibility through natural language instructions, but performance is sobering: NeuroLM-XL underperforms LaBraM-Huge on TUAB (0.797 vs. 0.826) and TUEV (0.468 vs. 0.662) despite 4.6× more parameters and 10× more data. Attributed to the causal attention constraint and the "domain alignment tax" of forcing EEG tokens into text embedding space. Zero independent benchmark evaluation of multimodal capabilities.

<h3>A.8 ZUNA (Warner et al., 2026)</h3>

The first diffusion-based EEG foundation model, trained on the largest corpus: approximately 2 million channel-hours from 208 datasets. 4D Rotary Positional Encoding encodes physical 3D electrode coordinates plus temporal index, enabling arbitrary electrode configurations. Architecture: 380M-parameter transformer encoder-decoder diffusion autoencoder with up to 90% channel dropout during training. Substantially outperforms spherical-spline interpolation for channel reconstruction, with advantage growing at higher dropout rates. However, evaluated exclusively on reconstruction and superresolution — never on downstream classification. Whether its encoder representations are useful for discriminative tasks remains the most obvious experiment no one has conducted.

<References items={[
  {
    "html": "Al-Shargie, F., Kiguchi, M., Badruddin, N., Dass, S. C., Hasan, A. F. M., and Tang, T. B. (2016). Mental stress assessment using simultaneous measurement of EEG and fNIRS. <em>Biomedical Optics Express</em>, 7(10), 3882-3898. <a href=\"https://doi.org/10.1364/BOE.7.003882\" target=\"_blank\" rel=\"noopener noreferrer\">https://doi.org/10.1364/BOE.7.003882</a>"
  },
  {
    "html": "Anthropic. (2025). Natural Emergent Misalignment from Reward Hacking in Production RL. Technical report. <a href=\"https://arxiv.org/abs/2511.18397\" target=\"_blank\" rel=\"noopener noreferrer\">https://arxiv.org/abs/2511.18397</a>"
  },
  {
    "html": "Arctop. (2023). Real-time cognitive state decoding from consumer-grade EEG. Technical report and product documentation. <a href=\"https://arctop.com\" target=\"_blank\" rel=\"noopener noreferrer\">https://arctop.com</a>"
  },
  {
    "html": "Babu, N., Mathew, J., and Vinod, A. P. (2025). Large Language Models for EEG: A Comprehensive Survey and Taxonomy. arXiv:2506.06353. <a href=\"https://arxiv.org/abs/2506.06353\" target=\"_blank\" rel=\"noopener noreferrer\">https://arxiv.org/abs/2506.06353</a>"
  },
  {
    "html": "Bilgin, I. P., St-Laurent, M., Bellec, L. P., and Wehbe, L. (2026). Brain-Informed Language Model Training Enables Scalable and Generalizable Alignment with Human Brain Activity. <a href=\"https://openreview.net/forum?id=07S1CPoQYP\" target=\"_blank\" rel=\"noopener noreferrer\">https://openreview.net/forum?id=07S1CPoQYP</a>"
  },
  {
    "html": "Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., et al. (2022). Constitutional AI: Harmlessness from AI Feedback. arXiv:2212.08073. <a href=\"https://arxiv.org/abs/2212.08073\" target=\"_blank\" rel=\"noopener noreferrer\">https://arxiv.org/abs/2212.08073</a>"
  },
  {
    "html": "Bjork, R. A. and Bjork, E. L. (2011). Making things hard on yourself, but in a good way: Creating desirable difficulties to enhance learning. In M. A. Gernsbacher, R. W. Pew, L. M. Hough, and J. R. Pomerantz (Eds.), <em>Psychology and the Real World: Essays Illustrating Fundamental Contributions to Society</em> (pp. 56-64). Worth Publishers. <a href=\"https://bjorklab.psych.ucla.edu/wp-content/uploads/sites/13/2016/04/EBjork_RBjork_2011.pdf\" target=\"_blank\" rel=\"noopener noreferrer\">https://bjorklab.psych.ucla.edu/wp-content/uploads/sites/13/2016/04/EBjork_RBjork_2011.pdf</a>"
  },
  {
    "html": "Baron-Cohen, S. (1995). <em>Mindblindness: An Essay on Autism and Theory of Mind.</em> MIT Press. <a href=\"https://mitpress.mit.edu/9780262522250/mindblindness/\" target=\"_blank\" rel=\"noopener noreferrer\">https://mitpress.mit.edu/9780262522250/mindblindness/</a>"
  },
  {
    "html": "Barrett-Lennard, G. T. (1962). Dimensions of therapist response as causal factors in therapeutic change. <em>Psychological Monographs</em>, 76(43), 1-36. <a href=\"https://doi.org/10.1037/h0093918\" target=\"_blank\" rel=\"noopener noreferrer\">https://doi.org/10.1037/h0093918</a>"
  },
  {
    "html": "Bickmore, T. W., Gruber, A., and Picard, R. W. (2005). Establishing the computer-patient working alliance in automated health behavior change interventions. <em>Patient Education and Counseling</em>, 59(1), 21-30. <a href=\"https://doi.org/10.1016/j.pec.2004.09.008\" target=\"_blank\" rel=\"noopener noreferrer\">https://doi.org/10.1016/j.pec.2004.09.008</a>"
  },
  {
    "html": "Calvo, R. A. and D'Mello, S. K. (2010). Affect Detection: An Interdisciplinary Review of Models, Methods, and Their Applications. <em>IEEE Transactions on Affective Computing</em>, 1(1), 18-37. <a href=\"https://doi.org/10.1109/T-AFFC.2010.1\" target=\"_blank\" rel=\"noopener noreferrer\">https://doi.org/10.1109/T-AFFC.2010.1</a>"
  },
  {
    "html": "Caria, A. (2025). Towards Predictive Communication: The Fusion of Large Language Models and Brain-Computer Interface. <em>Sensors</em>, 25(13), 3987. <a href=\"https://doi.org/10.3390/s25133987\" target=\"_blank\" rel=\"noopener noreferrer\">https://doi.org/10.3390/s25133987</a>"
  },
  {
    "html": "Chandrasekharan, S. and Jacob, J. E. (2025). Bridging neuroscience and AI: a survey on large language models for neurological signal interpretation. <em>Frontiers in Neuroinformatics</em>, 19, 1561401. <a href=\"https://doi.org/10.3389/fninf.2025.1561401\" target=\"_blank\" rel=\"noopener noreferrer\">https://doi.org/10.3389/fninf.2025.1561401</a>"
  },
  {
    "html": "Cheng, M., Lee, C., Khadpe, P., Yu, S., Han, D., &amp; Jurafsky, D. (2026). Sycophantic AI decreases prosocial intentions and promotes dependence. Science, 391(6792), aec8352."
  },
  {
    "html": "Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. (2017). Deep reinforcement learning from human preferences. <em>Advances in Neural Information Processing Systems</em>, 30. <a href=\"https://arxiv.org/abs/1706.03741\" target=\"_blank\" rel=\"noopener noreferrer\">https://arxiv.org/abs/1706.03741</a>"
  },
  {
    "html": "Cornejo-Plaza, M. I., Cippitani, R., and Pasquino, T. (2024). Chilean Supreme Court ruling on the protection of brain activity: neurorights, personal data protection, and neurodata. <em>Frontiers in Psychology</em>, 15, 1330439. <a href=\"https://doi.org/10.3389/fpsyg.2024.1330439\" target=\"_blank\" rel=\"noopener noreferrer\">https://doi.org/10.3389/fpsyg.2024.1330439</a>"
  },
  {
    "html": "Csikszentmihalyi, M. (1990). <em>Flow: The Psychology of Optimal Experience.</em> Harper &amp; Row. <a href=\"https://www.harpercollins.com/products/flow-mihaly-csikszentmihalyi\" target=\"_blank\" rel=\"noopener noreferrer\">https://www.harpercollins.com/products/flow-mihaly-csikszentmihalyi</a>"
  },
  {
    "html": "Cui, W., Jeong, W., Bhatt, J., Medina, B., Zhu, J., Li, B., et al. (2024). Neuro-GPT: Towards A Foundation Model for EEG. <em>2024 IEEE International Symposium on Biomedical Imaging (ISBI)</em>. <a href=\"https://arxiv.org/abs/2311.03764\" target=\"_blank\" rel=\"noopener noreferrer\">https://arxiv.org/abs/2311.03764</a>"
  },
  {
    "html": "D'Mello, S. K. and Graesser, A. C. (2012). AutoTutor and Affective AutoTutor: Learning by talking with cognitively and emotionally intelligent computers that talk back. <em>ACM Transactions on Interactive Intelligent Systems</em>, 2(4), 23:1-23:39. <a href=\"https://doi.org/10.1145/2395123.2395128\" target=\"_blank\" rel=\"noopener noreferrer\">https://doi.org/10.1145/2395123.2395128</a>"
  },
  {
    "html": "Davidson, R. J. (1992). Anterior cerebral asymmetry and the nature of emotion. <em>Brain and Cognition</em>, 20(1), 125-151. <a href=\"https://doi.org/10.1016/0278-2626(9\" target=\"_blank\" rel=\"noopener noreferrer\">https://doi.org/10.1016/0278-2626(9</a>2)90065-T"
  },
  {
    "html": "Davis, M. H. (1983). Measuring individual differences in empathy: Evidence for a multidimensional approach. <em>Journal of Personality and Social Psychology</em>, 44(1), 113-126. <a href=\"https://doi.org/10.1037/0022-3514.44.1.113\" target=\"_blank\" rel=\"noopener noreferrer\">https://doi.org/10.1037/0022-3514.44.1.113</a>"
  },
  {
    "html": "Ding, N. and Simon, J. Z. (2012). Emergence of neural encoding of auditory objects while listening to competing speakers. <em>Proceedings of the National Academy of Sciences</em>, 109(29), 11854-11859. <a href=\"https://doi.org/10.1073/pnas.1205381109\" target=\"_blank\" rel=\"noopener noreferrer\">https://doi.org/10.1073/pnas.1205381109</a>"
  },
  {
    "html": "Ethayarajh, K., Xu, W., Muennighoff, N., Jurafsky, D., and Kiela, D. (2024). KTO: Model Alignment as Prospect Theoretic Optimization. <em>International Conference on Machine Learning (ICML 2024)</em>. <a href=\"https://proceedings.mlr.press/v235/ethayarajh24a.html\" target=\"_blank\" rel=\"noopener noreferrer\">https://proceedings.mlr.press/v235/ethayarajh24a.html</a>"
  },
  {
    "html": "Eysenck, M. W., Derakshan, N., Santos, R., and Calvo, M. G. (2007). Anxiety and cognitive performance: Attentional control theory. <em>Emotion</em>, 7(2), 336-353. <a href=\"https://doi.org/10.1037/1528-3542.7.2.336\" target=\"_blank\" rel=\"noopener noreferrer\">https://doi.org/10.1037/1528-3542.7.2.336</a>"
  },
  {
    "html": "Fidencio, A. X., Grun, F., Klaes, C., and Iossifidis, I. (2025). Error-related Potential driven Reinforcement Learning for adaptive Brain-Computer Interfaces. arXiv:2502.18594. <a href=\"https://arxiv.org/abs/2502.18594\" target=\"_blank\" rel=\"noopener noreferrer\">https://arxiv.org/abs/2502.18594</a>"
  },
  {
    "html": "Fogg, B. J. (2003). <em>Persuasive Technology: Using Computers to Change What We Think and Do.</em> Morgan Kaufmann. <a href=\"https://doi.org/10.1016/B978-1-55860-643-2.X5000-8\" target=\"_blank\" rel=\"noopener noreferrer\">https://doi.org/10.1016/B978-1-55860-643-2.X5000-8</a>"
  },
  {
    "html": "Fu, J., Zhao, X., Yao, C., Wang, H., Han, Q., and Xiao, Y. (2025). Reward Shaping to Mitigate Reward Hacking in RLHF. arXiv:2502.18770. <a href=\"https://arxiv.org/abs/2502.18770\" target=\"_blank\" rel=\"noopener noreferrer\">https://arxiv.org/abs/2502.18770</a>"
  },
  {
    "html": "Furman, D. and Kwalwasser, E. (2021). Empathic Computing System and Methods for Improved Human Interactions With Digital Content Experiences. US Patent Application US20210390366A1, Arctop LTD. <a href=\"https://patents.google.com/patent/US20210390366A1\" target=\"_blank\" rel=\"noopener noreferrer\">https://patents.google.com/patent/US20210390366A1</a>"
  },
  {
    "html": "Gallese, V. and Goldman, A. (1998). Mirror neurons and the simulation theory of mind-reading. <em>Trends in Cognitive Sciences</em>, 2(12), 493-501. <a href=\"https://doi.org/10.1016/S1364-6613(9\" target=\"_blank\" rel=\"noopener noreferrer\">https://doi.org/10.1016/S1364-6613(9</a>8)01262-5"
  },
  {
    "html": "Gevins, A., Smith, M. E., McEvoy, L., and Yu, D. (1997). High-resolution EEG mapping of cortical activation related to working memory. <em>Cerebral Cortex</em>, 7(4), 374-385. <a href=\"https://doi.org/10.1093/cercor/7.4.374\" target=\"_blank\" rel=\"noopener noreferrer\">https://doi.org/10.1093/cercor/7.4.374</a>"
  },
  {
    "html": "Giraud, A.-L. and Poeppel, D. (2012). Cortical oscillations and speech processing: emerging computational principles and operations. <em>Nature Neuroscience</em>, 15(4), 511-517. <a href=\"https://doi.org/10.1038/nn.3063\" target=\"_blank\" rel=\"noopener noreferrer\">https://doi.org/10.1038/nn.3063</a>"
  },
  {
    "html": "Goldman, A. I. (2006). <em>Simulating Minds: The Philosophy, Psychology, and Neuroscience of Mindreading.</em> Oxford University Press. <a href=\"https://doi.org/10.1093/0195138929.001.0001\" target=\"_blank\" rel=\"noopener noreferrer\">https://doi.org/10.1093/0195138929.001.0001</a>"
  },
  {
    "html": "Gui, Y., Chen, M., Su, Y., Luo, G., and Yang, Y. (2025). EEGMamba: Bidirectional State Space Model with Mixture of Experts for EEG Multi-task Classification. arXiv:2407.20254. <a href=\"https://arxiv.org/abs/2407.20254\" target=\"_blank\" rel=\"noopener noreferrer\">https://arxiv.org/abs/2407.20254</a>"
  },
  {
    "html": "Guo, S., Zhang, B., Liu, T., Liu, T., Khalman, M., Llinares, F., et al. (2024). Direct Language Model Alignment from Online AI Feedback. arXiv:2402.04792. <a href=\"https://arxiv.org/abs/2402.04792\" target=\"_blank\" rel=\"noopener noreferrer\">https://arxiv.org/abs/2402.04792</a>"
  },
  {
    "html": "Hatcher, R. L. and Gillaspy, J. A. (2006). Development and validation of a revised short version of the Working Alliance Inventory. <em>Psychotherapy Research</em>, 16(1), 12-25. <a href=\"https://doi.org/10.1080/10503300500352500\" target=\"_blank\" rel=\"noopener noreferrer\">https://doi.org/10.1080/10503300500352500</a>"
  },
  {
    "html": "Hall, E. T. (1976). <em>Beyond Culture.</em> Anchor Books. <a href=\"https://www.penguinrandomhouse.com/books/53427/beyond-culture-by-edward-t-hall/\" target=\"_blank\" rel=\"noopener noreferrer\">https://www.penguinrandomhouse.com/books/53427/beyond-culture-by-edward-t-hall/</a>"
  },
  {
    "html": "Henrich, J., Heine, S. J., and Norenzayan, A. (2010). The weirdest people in the world? <em>Behavioral and Brain Sciences</em>, 33(2-3), 61-83. <a href=\"https://doi.org/10.1017/S0140525X0999152X\" target=\"_blank\" rel=\"noopener noreferrer\">https://doi.org/10.1017/S0140525X0999152X</a>"
  },
  {
    "html": "Horvath, A. O. and Symonds, B. D. (1991). Relation between working alliance and outcome in psychotherapy: A meta-analysis. <em>Journal of Counseling Psychology</em>, 38(2), 139-149. <a href=\"https://doi.org/10.1037/0022-0167.38.2.139\" target=\"_blank\" rel=\"noopener noreferrer\">https://doi.org/10.1037/0022-0167.38.2.139</a>"
  },
  {
    "html": "Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., et al. (2022). LoRA: Low-Rank Adaptation of Large Language Models. <em>International Conference on Learning Representations (ICLR 2022)</em>. <a href=\"https://arxiv.org/abs/2106.09685\" target=\"_blank\" rel=\"noopener noreferrer\">https://arxiv.org/abs/2106.09685</a>"
  },
  {
    "html": "Ienca, M. and Andorno, R. (2017). Towards new human rights in the age of neuroscience and neurotechnology. <em>Life Sciences, Society and Policy</em>, 13(1), 5. <a href=\"https://doi.org/10.1186/s40504-017-0050-1\" target=\"_blank\" rel=\"noopener noreferrer\">https://doi.org/10.1186/s40504-017-0050-1</a>"
  },
  {
    "html": "Jiang, W.-B., Zhao, L., and Lu, B.-L. (2024a). Large Brain Model for Learning Generic Representations with Tremendous EEG Data in BCI. <em>International Conference on Learning Representations (ICLR 2024)</em>. <a href=\"https://openreview.net/forum?id=QzTpTRVtrP\" target=\"_blank\" rel=\"noopener noreferrer\">https://openreview.net/forum?id=QzTpTRVtrP</a>"
  },
  {
    "html": "Jiang, W.-B., Wang, Y., Lu, B.-L., and Li, D. (2024b). NeuroLM: A Universal Multi-task Foundation Model for Bridging the Gap between Language and EEG Signals. arXiv:2409.00101. <a href=\"https://arxiv.org/abs/2409.00101\" target=\"_blank\" rel=\"noopener noreferrer\">https://arxiv.org/abs/2409.00101</a>"
  },
  {
    "html": "Kastrati, A., Bürki, J., Lauer, J., Xuan, C., Iaquinto, R., and Wattenhofer, R. (2025). EEG-Bench: A Benchmark for EEG Foundation Models in Clinical Applications. <em>Advances in Neural Information Processing Systems</em>, 38. <a href=\"https://arxiv.org/abs/2512.08959\" target=\"_blank\" rel=\"noopener noreferrer\">https://arxiv.org/abs/2512.08959</a>"
  },
  {
    "html": "Kamzanova, A. T., Kustubayeva, A. M., and Matthews, G. (2014). Use of EEG workload indices for diagnostic monitoring of vigilance decrement. <em>Human Factors</em>, 56(6), 1136-1149. <a href=\"https://doi.org/10.1177/0018720814525628\" target=\"_blank\" rel=\"noopener noreferrer\">https://doi.org/10.1177/0018720814525628</a>"
  },
  {
    "html": "Katahira, K., Yamazaki, Y., Yamaoka, C., Ozaki, H., Nakagawa, S., and Nagata, N. (2018). EEG correlates of the flow state. <em>Frontiers in Psychology</em>, 9, 300. <a href=\"https://doi.org/10.3389/fpsyg.2018.00300\" target=\"_blank\" rel=\"noopener noreferrer\">https://doi.org/10.3389/fpsyg.2018.00300</a>"
  },
  {
    "html": "Kirk, R., Mediratta, I., Nalmpantis, C., Luketina, J., Hambro, E., Grefenstette, E., et al. (2024). Understanding the Effects of RLHF on LLM Generalisation and Diversity. <em>International Conference on Learning Representations (ICLR 2024)</em>. <a href=\"https://openreview.net/forum?id=PXD3FAVHJT\" target=\"_blank\" rel=\"noopener noreferrer\">https://openreview.net/forum?id=PXD3FAVHJT</a>"
  },
  {
    "html": "Kostas, D., Aroca-Ouellette, S., and Rudzicz, F. (2021). BENDr: Using Transformers and a Contrastive Self-Supervised Learning Task to Learn From Massive Amounts of EEG Data. <em>Frontiers in Human Neuroscience</em>, 15, 653659. <a href=\"https://doi.org/10.3389/fnhum.2021.653659\" target=\"_blank\" rel=\"noopener noreferrer\">https://doi.org/10.3389/fnhum.2021.653659</a>"
  },
  {
    "html": "Koelstra, S., Muhl, C., Soleymani, M., Lee, J. S., Yazdani, A., Ebrahimi, T., et al. (2012). DEAP: A Database for Emotion Analysis using Physiological Signals. <em>IEEE Transactions on Affective Computing</em>, 3(1), 18-31. <a href=\"https://doi.org/10.1109/T-AFFC.2011.15\" target=\"_blank\" rel=\"noopener noreferrer\">https://doi.org/10.1109/T-AFFC.2011.15</a>"
  },
  {
    "html": "Kraskov, A., Stogbauer, H., and Grassberger, P. (2004). Estimating mutual information. <em>Physical Review E</em>, 69(6), 066138. <a href=\"https://doi.org/10.1103/PhysRevE.69.066138\" target=\"_blank\" rel=\"noopener noreferrer\">https://doi.org/10.1103/PhysRevE.69.066138</a>"
  },
  {
    "html": "Kuruppu, G., Wagh, N., Kremen, V., Pati, S., Worrell, G., and Varatharajah, Y. (2025). EEG Foundation Models: A Critical Review of Current Progress and Future Directions. arXiv:2507.11783. <a href=\"https://arxiv.org/abs/2507.11783\" target=\"_blank\" rel=\"noopener noreferrer\">https://arxiv.org/abs/2507.11783</a>"
  },
  {
    "html": "Lakens, D. (2017). Equivalence Tests: A Practical Primer for t Tests, Correlations, and Meta-Analyses. <em>Social Psychological and Personality Science</em>, 8(4), 355-362. <a href=\"https://doi.org/10.1177/1948550617697177\" target=\"_blank\" rel=\"noopener noreferrer\">https://doi.org/10.1177/1948550617697177</a>"
  },
  {
    "html": "Lee, H., Phatale, S., Mansoor, H., Mesnard, T., Ferret, J., Lu, K. R., et al. (2023). RLAIF: Scaling Reinforcement Learning from Human Feedback with AI Feedback. arXiv:2309.00267. <a href=\"https://arxiv.org/abs/2309.00267\" target=\"_blank\" rel=\"noopener noreferrer\">https://arxiv.org/abs/2309.00267</a>"
  },
  {
    "html": "Lopez-Paz, D., Bottou, L., Scholkopf, B., and Vapnik, V. (2016). Unifying distillation and privileged information. <em>International Conference on Learning Representations (ICLR 2016)</em>. <a href=\"https://arxiv.org/abs/1511.03643\" target=\"_blank\" rel=\"noopener noreferrer\">https://arxiv.org/abs/1511.03643</a>"
  },
  {
    "html": "Lotte, F., Bougrain, L., Cichocki, A., Clerc, M., Congedo, M., Rakotomamonjy, A., et al. (2018). A review of classification algorithms for EEG-based brain-computer interfaces: a 10 year update. <em>Journal of Neural Engineering</em>, 15(3), 031005. <a href=\"https://doi.org/10.1088/1741-2552/aab2f2\" target=\"_blank\" rel=\"noopener noreferrer\">https://doi.org/10.1088/1741-2552/aab2f2</a>"
  },
  {
    "html": "Mesgarani, N. and Chang, E. F. (2012). Selective cortical representation of attended speaker in multi-talker speech perception. <em>Nature</em>, 485(7397), 233-236. <a href=\"https://doi.org/10.1038/nature11020\" target=\"_blank\" rel=\"noopener noreferrer\">https://doi.org/10.1038/nature11020</a>"
  },
  {
    "html": "Mishra, A., Shukla, S., Torres, J., Gwizdka, J., and Roychowdhury, S. (2024). Thought2Text: Text Generation from EEG Signal using Large Language Models. arXiv:2410.07507. <a href=\"https://arxiv.org/abs/2410.07507\" target=\"_blank\" rel=\"noopener noreferrer\">https://arxiv.org/abs/2410.07507</a>"
  },
  {
    "html": "Muhl, C., Allison, B., Nijholt, A., and Chanel, G. (2014). A survey of affective brain computer interfaces. <em>Brain-Computer Interfaces</em>, 1(2), 66-84. <a href=\"https://doi.org/10.1080/2326263X.2014.912881\" target=\"_blank\" rel=\"noopener noreferrer\">https://doi.org/10.1080/2326263X.2014.912881</a>"
  },
  {
    "html": "Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., et al. (2022). Training language models to follow instructions with human feedback. <em>Advances in Neural Information Processing Systems</em>, 35. <a href=\"https://arxiv.org/abs/2203.02155\" target=\"_blank\" rel=\"noopener noreferrer\">https://arxiv.org/abs/2203.02155</a>"
  },
  {
    "html": "Pearl, J. (2009). <em>Causality: Models, Reasoning, and Inference.</em> 2nd edition. Cambridge University Press. <a href=\"https://doi.org/10.1017/CBO9780511803161\" target=\"_blank\" rel=\"noopener noreferrer\">https://doi.org/10.1017/CBO9780511803161</a>"
  },
  {
    "html": "Picard, R. W. (1997). <em>Affective Computing.</em> MIT Press. <a href=\"https://mitpress.mit.edu/9780262661157/affective-computing/\" target=\"_blank\" rel=\"noopener noreferrer\">https://mitpress.mit.edu/9780262661157/affective-computing/</a>"
  },
  {
    "html": "Premack, D. and Woodruff, G. (1978). Does the chimpanzee have a theory of mind? <em>Behavioral and Brain Sciences</em>, 1(4), 515-526. <a href=\"https://doi.org/10.1017/S0140525X00076512\" target=\"_blank\" rel=\"noopener noreferrer\">https://doi.org/10.1017/S0140525X00076512</a>"
  },
  {
    "html": "Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. <em>Advances in Neural Information Processing Systems</em>, 36. <a href=\"https://arxiv.org/abs/2305.18290\" target=\"_blank\" rel=\"noopener noreferrer\">https://arxiv.org/abs/2305.18290</a>"
  },
  {
    "html": "Rashkin, H., Smith, E. M., Li, M., and Boureau, Y.-L. (2019). Towards Empathetic Open-domain Conversation Models: A New Benchmark and Dataset. <em>Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL 2019)</em>, 5370-5381. <a href=\"https://aclanthology.org/P19-1534/\" target=\"_blank\" rel=\"noopener noreferrer\">https://aclanthology.org/P19-1534/</a>"
  },
  {
    "html": "Risko, E. F. and Gilbert, S. J. (2016). Cognitive Offloading. <em>Trends in Cognitive Sciences</em>, 20(9), 676-688. <a href=\"https://doi.org/10.1016/j.tics.2016.07.002\" target=\"_blank\" rel=\"noopener noreferrer\">https://doi.org/10.1016/j.tics.2016.07.002</a>"
  },
  {
    "html": "Rouder, J. N., Speckman, P. L., Sun, D., Morey, R. D., and Iverson, G. (2009). Bayesian t tests for accepting and rejecting the null hypothesis. <em>Psychonomic Bulletin &amp; Review</em>, 16(2), 225-237. <a href=\"https://doi.org/10.3758/PBR.16.2.225\" target=\"_blank\" rel=\"noopener noreferrer\">https://doi.org/10.3758/PBR.16.2.225</a>"
  },
  {
    "html": "Saby, J. N. and Marshall, P. J. (2012). The Utility of EEG Band Power Analysis in the Study of Infancy and Early Childhood. <em>Developmental Neuropsychology</em>, 37(3), 253-273. <a href=\"https://doi.org/10.1080/87565641.2011.614663\" target=\"_blank\" rel=\"noopener noreferrer\">https://doi.org/10.1080/87565641.2011.614663</a>"
  },
  {
    "html": "Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017). Proximal Policy Optimization Algorithms. arXiv:1707.06347. <a href=\"https://arxiv.org/abs/1707.06347\" target=\"_blank\" rel=\"noopener noreferrer\">https://arxiv.org/abs/1707.06347</a>"
  },
  {
    "html": "Schull, N. D. (2012). <em>Addiction by Design: Machine Gambling in Las Vegas.</em> Princeton University Press. <a href=\"https://press.princeton.edu/books/paperback/9780691160887/addiction-by-design\" target=\"_blank\" rel=\"noopener noreferrer\">https://press.princeton.edu/books/paperback/9780691160887/addiction-by-design</a>"
  },
  {
    "html": "Sharma, M., Tong, M., Korbak, T., Duvenaud, D., Askell, A., Bowman, S. R., et al. (2024). Towards Understanding Sycophancy in Language Models. <em>International Conference on Learning Representations (ICLR 2024)</em>. <a href=\"https://openreview.net/forum?id=tvhaxkMKAn\" target=\"_blank\" rel=\"noopener noreferrer\">https://openreview.net/forum?id=tvhaxkMKAn</a>"
  },
  {
    "html": "Sitaram, R., Ros, T., Stoeckel, L., Haller, S., Scharnowski, F., Lewis-Peacock, J., et al. (2017). Closed-loop brain training: the science of neurofeedback. <em>Nature Reviews Neuroscience</em>, 18(2), 86-100. <a href=\"https://doi.org/10.1038/nrn.2016.164\" target=\"_blank\" rel=\"noopener noreferrer\">https://doi.org/10.1038/nrn.2016.164</a>"
  },
  {
    "html": "Sorino, P., Biancofiore, G. M., Lofu, D., Colafiglio, T., Lombardi, A., Narducci, F., et al. (2024). ARIEL: Brain-Computer Interfaces meet Large Language Models for Emotional Support Conversation. <em>Adjunct Proceedings of the 32nd ACM Conference on User Modeling, Adaptation and Personalization (UMAP Adjunct '24)</em>, Cagliari, Italy. <a href=\"https://doi.org/10.1145/3631700.3665193\" target=\"_blank\" rel=\"noopener noreferrer\">https://doi.org/10.1145/3631700.3665193</a>"
  },
  {
    "html": "Stiennon, N., Ouyang, L., Wu, J., Ziegler, D., Lowe, R., Voss, C., et al. (2020). Learning to summarize from human feedback. <em>Advances in Neural Information Processing Systems</em>, 33. <a href=\"https://arxiv.org/abs/2009.01325\" target=\"_blank\" rel=\"noopener noreferrer\">https://arxiv.org/abs/2009.01325</a>"
  },
  {
    "html": "Sweller, J. (1988). Cognitive load during problem solving: Effects on learning. <em>Cognitive Science</em>, 12(2), 257-285. <a href=\"https://doi.org/10.1207/s15516709cog1202_4\" target=\"_blank\" rel=\"noopener noreferrer\">https://doi.org/10.1207/s15516709cog1202_4</a>"
  },
  {
    "html": "Ullman, T. (2023). Large Language Models Fail on Trivial Alterations to Theory-of-Mind Tasks. arXiv:2302.08399. <a href=\"https://arxiv.org/abs/2302.08399\" target=\"_blank\" rel=\"noopener noreferrer\">https://arxiv.org/abs/2302.08399</a>"
  },
  {
    "html": "UNESCO. (2025). Recommendation on the ethics of neurotechnology. <a href=\"https://www.unesco.org/en/ethics-neurotech/recommendation\" target=\"_blank\" rel=\"noopener noreferrer\">https://www.unesco.org/en/ethics-neurotech/recommendation</a>"
  },
  {
    "html": "Vapnik, V. and Vashist, A. (2009). A new learning paradigm: Learning using privileged information. <em>Neural Networks</em>, 22(5-6), 544-557. <a href=\"https://doi.org/10.1016/j.neunet.2009.06.042\" target=\"_blank\" rel=\"noopener noreferrer\">https://doi.org/10.1016/j.neunet.2009.06.042</a>"
  },
  {
    "html": "Wang, G., Liu, W., He, Y., Xu, C., Ma, L., and Li, H. (2024). EEGPT: Pretrained Transformer for Universal and Reliable Representation of EEG Signals. <em>Advances in Neural Information Processing Systems</em>, 37. <a href=\"https://proceedings.neurips.cc/paper_files/paper/2024/hash/4540d267eeec4e5dbd9dae9448f0b739-Abstract-Conference.html\" target=\"_blank\" rel=\"noopener noreferrer\">https://proceedings.neurips.cc/paper_files/paper/2024/hash/4540d267eeec4e5dbd9dae9448f0b739-Abstract-Conference.html</a>"
  },
  {
    "html": "Wang, J., Zhao, S., Luo, Z., Zhou, Y., Jiang, H., Li, S., et al. (2025). CBraMod: A Criss-Cross Brain Foundation Model for EEG Decoding. <em>International Conference on Learning Representations (ICLR 2025)</em>. <a href=\"https://openreview.net/forum?id=NPNUHgHF2w\" target=\"_blank\" rel=\"noopener noreferrer\">https://openreview.net/forum?id=NPNUHgHF2w</a>"
  },
  {
    "html": "Warner, C., Mago, J., Huml, J. R., Osman, M., and Millidge, B. (2026). ZUNA: Flexible EEG Superresolution with Position-Aware Diffusion Autoencoders. arXiv:2602.18478. <a href=\"https://arxiv.org/abs/2602.18478\" target=\"_blank\" rel=\"noopener noreferrer\">https://arxiv.org/abs/2602.18478</a>"
  },
  {
    "html": "Wobrock, D., Frey, J., Graeff, D., de la Riviere, J.-B., Castet, J., and Lotte, F. (2015). Continuous Mental Effort Evaluation during 3D Object Manipulation Tasks based on Brain and Physiological Signals. <em>INTERACT 2015</em>, Lecture Notes in Computer Science, vol. 9297, 472-487. <a href=\"https://doi.org/10.1007/978-3-319-22701-6_35\" target=\"_blank\" rel=\"noopener noreferrer\">https://doi.org/10.1007/978-3-319-22701-6_35</a>"
  },
  {
    "html": "Wolpaw, J. R., Birbaumer, N., McFarland, D. J., Pfurtscheller, G., and Vaughan, T. M. (2002). Brain-computer interfaces for communication and control. <em>Clinical Neurophysiology</em>, 113(6), 767-791. <a href=\"https://doi.org/10.1016/S1388-2457(0\" target=\"_blank\" rel=\"noopener noreferrer\">https://doi.org/10.1016/S1388-2457(0</a>2)00057-3"
  },
  {
    "html": "Xiong, W., Li, J., Li, J., Zhu, K., and Jiang, C. (2025). EEG-FM-Bench: A Comprehensive Benchmark for the Systematic Evaluation of EEG Foundation Models. arXiv:2508.17742. <a href=\"https://arxiv.org/abs/2508.17742\" target=\"_blank\" rel=\"noopener noreferrer\">https://arxiv.org/abs/2508.17742</a>"
  },
  {
    "html": "Xu, S., Fu, W., Gao, J., Ye, W., Liu, W., Mei, Z., et al. (2024). Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study. <em>International Conference on Machine Learning (ICML 2024)</em>. <a href=\"https://proceedings.mlr.press/v235/xu24h.html\" target=\"_blank\" rel=\"noopener noreferrer\">https://proceedings.mlr.press/v235/xu24h.html</a>"
  },
  {
    "html": "Yang, C., Westover, M. B., and Sun, J. (2023). BIOT: Biosignal Transformer for Cross-data Learning in the Wild. <em>Advances in Neural Information Processing Systems</em>, 36. <a href=\"https://openreview.net/forum?id=c2LZyTyddi\" target=\"_blank\" rel=\"noopener noreferrer\">https://openreview.net/forum?id=c2LZyTyddi</a>"
  },
  {
    "html": "Yang, L., Sun, Q., Li, A., and Van Hulle, M. M. (2026). Are EEG Foundation Models Worth It? Comparative Evaluation with Traditional Decoders in Diverse BCI Tasks. <em>International Conference on Learning Representations (ICLR 2026)</em>. <a href=\"https://openreview.net/forum?id=5Xwm8e6vbh\" target=\"_blank\" rel=\"noopener noreferrer\">https://openreview.net/forum?id=5Xwm8e6vbh</a>"
  },
  {
    "html": "Yuste, R., Goering, S., Agüera y Arcas, B., Bi, G., Carmena, J. M., Carter, A., et al. (2017). Four ethical priorities for neurotechnologies and AI. <em>Nature</em>, 551(7679), 159-163. <a href=\"https://doi.org/10.1038/551159a\" target=\"_blank\" rel=\"noopener noreferrer\">https://doi.org/10.1038/551159a</a>"
  },
  {
    "html": "Zander, T. O. and Kothe, C. (2011). Towards passive brain-computer interfaces. <em>Journal of Neural Engineering</em>, 8(2), 025005. <a href=\"https://doi.org/10.1088/1741-2560/8/2/025005\" target=\"_blank\" rel=\"noopener noreferrer\">https://doi.org/10.1088/1741-2560/8/2/025005</a>"
  },
  {
    "html": "Zhang, J., Shen, J., Tu, W., Zhang, Y., Zhang, H., Gedeon, T., et al. (2026). EEG-Based Brain-LLM Interface for Human Preference Aligned Generation. arXiv:2603.16897. <a href=\"https://arxiv.org/abs/2603.16897\" target=\"_blank\" rel=\"noopener noreferrer\">https://arxiv.org/abs/2603.16897</a>"
  },
  {
    "html": "Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., et al. (2019). Fine-Tuning Language Models from Human Preferences. arXiv:1909.08593. <a href=\"https://arxiv.org/abs/1909.08593\" target=\"_blank\" rel=\"noopener noreferrer\">https://arxiv.org/abs/1909.08593</a>"
  }
]} />

---

# Why Consumer EEG Embedded Hardware Is the Next Big Platform: My Predictions as a CTO
URL: https://arctop.com/deep-dives/consumer-eeg-hardware
Published: 2025-03-28

<h2>Introduction: A New Wave of Technology in Everyday Life</h2>


I believe we're on the brink of a new platform revolution in consumer tech: 
<strong>brain sensor-embedded wearable devices.</strong> Specifically, EEG sensors
will be embedded in devices at large scale, providing a platform for developing new 
services on, much like how GPS, accelerometers, and cameras went from novel extras to 
must-have sensors in every device. Why the excitement for a few new sensors? Because 
they unlock something fundamentally new and unprecedentedly powerful: the ability for 
our devices to sense cognitive state directly from the brain of users live in the moment. 
Fetching that class of emotional, cognitive data enables frictionless and powerful new input
methods to computers, opening up a new frontier of possibilities.  In this post, I'll explain
why I predict consumer EEG hardware will be the <strong>next big platform</strong>, and how it
will power smarter, more adaptive experiences in the very near future.


<h2>From GPS to Brainwaves: A Familiar Pattern</h2>


<strong>Brainwave-sensing hardware</strong> is poised to go mainstream by 2025–2026. As a CTO in this field, 
I'm seeing clear signs everywhere that what we predicted years ago is finally becoming reality.
<strong>Hardware developers are energized</strong> – they're gearing up to embed miniature, dry EEG
sensors into headphones, glasses, VR headsets, and more at scale – so app developers can build with
next level capabilities, making <strong>App developers psyched</strong> also which will lead to an 
abundance of amazing experiences that will drive network effects of consumer adoption.




Not long ago, you may recall, it was hard to imagine that every phone would have a GPS chip or a 
gyroscope. But as history shows, once a sensor becomes small and cheap enough to integrate en masse, 
it can rapidly turn into a platform for innovation. <strong>Smartphones today pack a suite of embedded 
sensors (accelerometers, compasses, gyroscopes, GPS, microphones, cameras, etc.) that enable countless 
applications across search, mobility, health, gaming, education and beyond.</strong> Developers ran with
those capabilities – giving us turn-by-turn directions, fitness trackers, AR games, and social media filters
that we now take for granted.




I see EEG sensors following a similar trajectory. Early brainwave gadgets—like research-grade headsets or 
niche meditation bands—were isolated devices suited only to specific contexts. At worst, they were 
'garbage-in-garbage-out' (“GIGO”) devices, technically capable of capturing EEG but with so much noise and 
unreliability that extracting meaningful data for computer interfaces or health insights was nearly impossible. 
But now high-quality EEG is breaking out of the lab, and the AI revolution in software is ready to embrace the 
new wave of EEG devices as optimized interfaces for daily actions like interactions with AI agents, fitness and health. 
Daily is the key here, like a toothbrush - most people will use the technology every day as part of their routine and it 
will get woven into much of life. We're now seeing brain sensors embedded directly into everyday consumer products — 
<strong>earbuds, headphones, AR/VR headsets, even wearable headbands and luxurious 
sleep masks.</strong>




The leap from a specialty device to a common smartphone accessory is the game-changer around the corner. When 
EEG hardware is standard on consumer devices, it becomes a <strong>platform</strong> for developers to build on, 
just like GPS did for location-based apps. In other words, brain-computer interface tech won't be confined to 
research or medical implants any longer; it will be in the hands (and ears) of everyday users.


<h2>Why EEG Wearables Are Set to Go Mainstream by 2025</h2>


Several converging factors make me confident that <strong>2025 is the tipping point for EEG wearables:</strong>


<ul>
<li>
<strong>Maturing Technology:</strong>
Consumer hardware has finally caught up to the vision. EEG sensors are now compact and power-efficient enough 
to tuck into normal-looking peripherals. For example, one newly released set of premium headphones from Master & Dynamic hides 
multiple EEG electrodes in the ear cushions, while looking and feeling like regular over-ear headphones. The 
difference is literally imperceptible to the wearer, but what the software can do for them if they are wearing 
the EEG one versus the non-embedded one is night and day.


<Figure src="/deep-dives/pics/deep-dives/consumer_eeg_hardware_pic1.webp" alt="MW75-Neuro headset with EEG electrodes in the ear cushions" caption={"<strong>MW75-Neuro headset with EEG electrodes in the ear cushions</strong>"} sourceUrl="https://www.masterdynamic.com/products/mw75-neuro" sourceLabel="Master & Dynamic" />




At CES 2025 we also saw <strong>brain-sensing earbuds</strong>  debuting from a few leading companies – sleek 
wireless buds that measure EEG from inside the ear canal and surrounding areas.




<Figure src="/deep-dives/pics/deep-dives/consumer_eeg_hardware_pic2.webp" alt="MW75-Neuro headset with EEG electrodes in the ear cushions" caption={"This is a far cry from the bulky research rigs of the past."} sourceUrl="https://mytone.io/" sourceLabel="Tone" />


</li>
<li>
<strong>Developer & Industry Enthusiasm:</strong>
As Arctop's CTO, I have the privilege of speaking with many hardware makers and product teams. 
The excitement in these conversations is palpable. Companies across the spectrum – from audio tech 
brands to AR/VR innovators – are actively at some stage of planning, prototyping or deploying 
<strong>EEG-enabled devices.</strong> We're talking smart headphone manufacturers adding health-sensing 
features, AR glasses startups looking to integrate EEG for context awareness and brain ID as passwordless 
login, and VR headset teams exploring cognitive inputs for mental health apps. Even major tech companies 
have signaled interest – industry trends show that big players are <em>“increasingly exploring physiological 
sensors as user inputs for AR/VR.”</em> AI companies also are assessing how and where BCI technology can be used
to give them an edge, or at the least - to avoid strategic surprise. In short, the personal computing hardware industry
is betting that brain-sensing is the next frontier, and the giants don't want to be disrupted and left behind.
</li>
<li>
<strong>User Demand for Deeper Insights:</strong>
The timing is right from a user perspective too. Over the past few years, consumers have grown comfortable with wearables 
that track heart rate, sleep, stress, and more. There's a growing cultural emphasis on mindfulness, focus, and mental well-being. 
People are asking for tech that not only counts steps, but also helps them understand and improve their <strong>cognitive and 
emotional state.</strong> EEG data fits perfectly into this trend. Imagine being able to see when you are truly focused versus 
mentally fatigued, or to objectively measure how calm your mind is during meditation. Or even better, let your 'Calm' app measure 
how calm you are in real-time and do whatever it can to increase that calmness uniquely for you. That kind of insight was only 
available in labs – now it's on the cusp of being available on your <strong>headphones or earbuds at home.</strong> Early evidence 
of demand can be seen in the success of products like the Muse meditation headband and the interest around Emotiv's new EEG earbuds. 
Users are excited to get their brain “vitals” in the same way they've come to expect heart-rate or steps on their smartwatch. 
They're curious to track themselves for self-knowledge, and also for actionable "SOS" early and emergency warning signals for 
themselves, their parents, kids or friends. Basically they want to learn and grow with the help of their personal biometric data 
in a variety of contexts, like how Whoop and Oura users came to love their new devices and build upon what began as a health trend 
of people tracking their steps.
</li>
<li>
<strong>Better Algorithms & SDKs:</strong>
Raw EEG signals are extremely complex, but here's the good news—developers don't need a neuroscience Ph.D. to make sense of them, 
just as they don't need a physics Ph.D. to implement GPS-based location features. Advances in machine learning and 
signal processing (including work by my team at Arctop) have made it much easier to interpret brainwave data in real time. We now 
have software that can detect patterns in EEG corresponding to recognizable mental states (cognitive workload, enjoyment, auditory 
attention, etc.) with surprising reliability. As an industry, we've finally created the abstraction layers needed to shield the gory 
neuroscience details from product developers and give them only what they need. (For example, <strong>our SDK at Arctop makes 
integrating cognitive state sensing plug-and-play,</strong> so developers can get meaningful metrics like “focus level” or “enjoyment level” 
derived from a device's EEG with just a few lines of code – no need to reinvent signal processing from scratch, let alone reinvent it 
for every compatible hardware device and platform). This means when EEG hardware lands in developers' hands, they can immediately 
start building cool apps and features that will be interoperable with multiple devices and application platforms instead of spending years 
researching brainwaves. The tooling is now at a point where <strong>cognitive signal integration is as straightforward as tapping into an 
accelerometer or camera feed.</strong> This is major for accelerating adoption.
</li>
</ul>

<h2>Unlocking a New Dimension: Cognitive State Sensing</h2>


The most profound reason EEG hardware is exciting is that it <strong>unlocks a new dimension of user understanding.</strong> At consumer scale, 
that means fundamentally new experiences for humans around the globe on a daily basis. Our devices have always been good at sensing the external 
world for us or our physical state – location, motion, touch, sound, vision, heart rate, etc. But they've been <strong>blind to what's happening inside us.</strong>
Inside that mush in our heads between our ears is a lot of electrical activity that is key to how we experience the world, to say the least. 
Finally that mush is being measured accurately and understood, and beyond that - being connected to computer understanding of user intention 
and perception. Consumer EEG, in other words, changes the computing paradigm. For the first time, mainstream gadgets will be able to capture 
aspects of our <em>cognitive</em> and <em>emotional</em> state in real time.




Why is this a big deal? Because having access to the user's cognitive state lets devices behave in ways that are far more <strong>context-aware, personalized, and empathetic.</strong>
I'm talking about software that adapts not only to your physical environment ie. where you are, but to <em>you, the individual user, who you are in that very moment.</em>
This extra layer of understanding – sometimes called <strong>cognition-aware design, or human-centric</strong> – is poised to revolutionize user experience. 
It's like going from 2D black-and-white to 3D full spectrum color. Suddenly apps can respond to whether you're confused or confident, overwhelmed or bored, 
angry or delighted. Developers can design interactions that flex to the user's <strong>mental state</strong> on the fly which makes technology feel significantly 
more humane and responsive. In fact, we've already seen early research and products hint at what's possible: everything from personalized learning systems to assistive 
communication devices to emotion-adaptive games powered by real-time brain data and interpretations on top of that electrophysiological continuous flux. 
I believe that as this capability becomes widely available, we'll look back and wonder how our computers ever functioned without understanding our state of mind.





<Figure src="/deep-dives/pics/deep-dives/consumer_eeg_hardware_pic3.webp" alt="another picture of the MW75-Neuro headset with EEG electrodes in the ear cushions" caption={"Smart headphones like the M&D MW75 Neuro are early examples of brain-sensing wearables. These devices use EEG sensors camouflaged in the ear pads to monitor the listener's cognitive state (e.g., focus levels) and adapt the experience accordingly"} sourceUrl="https://www.techradar.com/audio/headphones/i-tried-the-neuro-hacking-headphones-and-if-you-struggle-to-quieten-your-mind-you-should-too" sourceLabel="techradar.com" />



<h2>Smarter, More Adaptive Experiences Ahead</h2>


What kinds of new experiences can cognition-aware devices enable? The possibilities span many domains and again, similar to GPS-enabled location apps, 
the creativity of developers to create unpredictable viral hits cannot be underestimated. But let's explore a few clear and compelling use cases that 
developers and product teams are already dreaming up for <strong>user state-aware devices</strong> equipped with embedded EEG:


<ul>
<li>
<strong>Adaptive Audio Headphones:</strong>
Imagine your smart headphones detecting that your mind is wandering or your stress is spiking, and automatically adjusting the audio to help. 
If you're distracted at work, the headphones could dial up noise cancellation or switch to an instrumental playlist to improve your focus. 
If they sense you're stressed, they might play calming ambient sounds or gentle biofeedback tones to steady your nerves. The result is an 
<strong>adaptive audio experience</strong> that tunes itself to your mental state in real time.
</li>
<li>
<strong>Immersion-Aware AR/VR Headsets:</strong>
Next-gen mixed reality headsets (think Vision Pro–style devices) will likely incorporate EEG to gauge your mental load and fatigue. 
Such a headset could detect when you're getting overwhelmed or tired during an intense VR session and dynamically adjust the experience 
for example through the immersion level. If cognitive fatigue is detected, the system might dim the brightness, simplify the visuals, or 
reduce the immersion intensity to maintain comfort. It could even suggest a short break or switch to a more passive content mode when your 
focus drops. <strong>Immersion level becomes fluid,</strong> automatically tuning up when you're engaged and scaling back when your brain needs a breather. 
This would keep extended AR/VR use comfortable and personalized, preventing tech burnout by respecting the user's cognitive limits.
</li>
<li>
<strong>Engagement-Adaptive Learning:</strong>
One of the areas I'm most excited about is education. E-learning platforms and training apps can become dramatically more effective if they know the 
learner's engagement and cognitive state. With EEG sensors in a pair of study headphones or AR glasses, a learning app could detect if you're 
truly absorbing the material or if your attention is fading. If the system senses high engagement and flow, it could capitalize on that moment – 
maybe increase the challenge or speed through easier content. If it detects confusion or mental fatigue, it could pause to review the concept in 
a different way, or insert a short interactive to re-capture your attention. The pacing and difficulty adapt continuously to <strong>maximize the learner's 
information retention</strong> and <strong>minimize confusion.</strong> This kind of cognition-aware tutoring could help keep students out of the dreaded 
boredom or overload zones, optimizing learning efficiency. It's like having a tutor who can read your mind and personalize the lesson instantaneously.
</li>
</ul>


These scenarios are just the tip of the iceberg. Virtually any experience that can benefit from understanding how the user is feeling or responding in the 
moment could tap into brain signals to become more adaptive. Gaming, for instance, might adjust difficulty based on a player's stress or focus. 
Fitness apps could use brain data to find the sweet spot between pushing you and not overstressing you. Even automotive interfaces might monitor 
driver alertness to improve safety. The common theme is that <strong>devices will no longer be oblivious to the user's mindstate</strong> – and that 
opens up an incredible design space for innovation.



<h2>Enabling Developers: Lowering the Barrier with Plug-and-Play Neuroscience</h2>


One question I often get is, “This sounds awesome, but how can our team actually implement brain-sensing features without neuroscience expertise?” 
This is where I gently highlight the role of platforms like ours. At <strong>Arctop,</strong> our mission has been to make brain-computer interface technology 
accessible to developers. In practice, that means we provide an SDK and APIs that handle the heavy lifting of EEG signal processing and interpretation. 
We've spent years decoding the language of the brain so that you, as a developer or product designer, don't have to.




The goal is to make integrating cognitive signals as easy as any other sensor. You get high-level metrics (for example: “user's cognitive workload = 70%”) 
that your software can use to adapt the experience. Meanwhile, under the hood our platform is doing the complex work of filtering EEG noise, 
running machine learning models, and translating neural patterns into those useful insights. By abstracting away the neuroscience, we let product teams 
focus on what they do best – building creative applications and delightful user experiences. It's similar to how early smartphone app developers didn't 
need to know the physics of GPS satellites; they just called an API to get location and then built cool services on top of it. We envision EEG data working 
the same way: request cognitive state info, get a stream of values/events, and build with it. This plug-and-play approach is already helping developers 
experiment with cognition-aware features without a steep learning curve.




I mention Arctop's role not to pat ourselves on the back, but to reassure the tech community that <strong>the tools are ready</strong> for this next big platform. 
The industry (including us and others) has been laying the groundwork so that when the new EEG-enabled hardware hits the market, developers can 
hit the ground running. We want to fast-forward to the creative part – seeing what amazing new applications you all will invent when devices truly 
understand users at the cognitive level.


<h2>Conclusion: Cognition-Aware Design is the Future</h2>


In my view, <strong>cognition-aware design</strong> represents the next evolution in human-computer interaction. We've made our devices context-aware about our external world; 
next they will become <strong>context-aware about us</strong> – our focus, enjoyment, interest, and more. Consumer EEG hardware is the key that unlocks this capability, 
and its impending mainstream arrival is why I'm so confident it's the next big platform. The years 2025–2026 will likely be remembered as the time 
when neurotechnology quietly slipped into everyone's daily gadgets.




For developers, product teams, and forward-looking technologists, the message is clear: now is the time to start thinking about how <strong>brain-computer interface</strong>
capabilities can enhance your products. The hardware is coming – in some cases, it's already here – and the demand for more intuitive, personalized, and responsive 
tech experiences is only growing. Early adopters who build <strong>user state-aware applications</strong> are going to shape the next generation of tech user experience, just like the 
first movers in mobile apps did with location services or accelerometer-based interactions. Investors and innovators take note as well: an ecosystem of EEG-enabled 
wearables and apps is emerging, with opportunities to lead in new market categories (<strong>EEG wearables, cognition-aware apps, adaptive audio, brain-health tech,</strong> and so on).




At Arctop, we're thrilled to be enabling this trend in our own way, but it's the broader industry and developer community that will truly bring it to life. I've focused my 
career on brain-computer interfaces because I genuinely believe this technology will make technology more human-centric. When our devices can sense and respect our cognitive 
state, they can collaborate with us more intelligently. They can nudge us to be our best, or help us when we're not at our best. They can make experiences feel magically <strong>tailored 
in the moment,</strong> which is ultimately what users crave – technology that “gets” them.




Thank you for reading this deep dive into the future of consumer EEG hardware. I hope it inspires you to think about how this technology will transform the way we interact with technology.

---

# Memory Decoding at Home: Mapping Inter-Subject Neural Synchrony to Episodic Memory Performance
URL: https://arctop.com/deep-dives/memory-decoding-from-eeg-at-home
Published: 2024-07-15

This post is meant as an informal, quick transmission of a technical
discovery to introduce our novel algorithm (FCCA) for mapping
inter-subject neural synchrony to memory function. Fuller description of
methods and discussion to follow in an upcoming version. The authors
would like to thank collaborators Erez Simony, Aia Haruvi, Shai Kalev,
Ronen Kopito for their contributions.


<h2>Abstract</h2>


Neural synchrony, similarities in brain activity across individuals,
offers valuable insights into emotional and cognitive processing. It can
also be used, under certain conditions, to infer subjective states of
consciousness<sup>1</sup>. Objective metrics of similarities between
people's brain data, or inter-subject correlations (ISC), we map here to
dynamics of memory performance using a novel algorithm that builds upon
the framework developed in correlated components analysis (CorrCA),
generating a new higher-order analysis layer that obtains superior
performance in memory decoding from EEG data compared to all other known
methods. We call the method feature-based CorrCA (FCCA), and find it
enables a robust, accurate memory analysis from people at home, in
non-laboratory environments, involved in everyday tasks. By selectively
deriving and using particular EEG activity features relevant to
cognitive and emotional processes reflected in neural synchrony, FCCA
serves as a powerful tool for investigation of the neural basis of
shared experiences and highlights the nuance needed for accurate memory
decoding from EEG. Validated on two models of consumer-grade EEG
recording devices, our generalizable approach offers new possibilities
for studying collective experiences in various contexts, from education
to entertainment, and contributes to the study and understanding of
social cognition and memory.



Episodic memory in particular has been an active area of research for
modern neuroscience and there is a wealth of published findings in the
literature relating to how physiological measures of brain activity
during information encoding map to behaviorally demonstrable memory
performance in the future. (Lee 2022)(Rugg, 2007, 2013), (Simony 2016),
(Summerfield 2006). Studying hippocampus-cortex interaction during movie
watching has been one fruitful experimental direction for the field.
Along this line of inquiry we report on a naturalistic neuroscience
study we conducted involving 60 adult participants distributed across
the United States. Participants measured their EEG brain signal using a
consumer wearable headband with sensors embedded while watching 38 short
video clips and a 27-minute episode of the TV show 'Curb Your
Enthusiasm' and days later were quizzed through multiple choice tests on
aspects of the content to investigate whether FCCA can help elucidate
relationships between memory formation and neural synchrony. We found
that ISC dynamics derived by our FCCA algorithm can effectively capture
memorability level dynamics while individuals experience the same
stimulus.


<h2>Introduction</h2>


Earlier research has established a significant relationship between
inter-subject neural synchrony and memory<sup>2,3</sup>. Both
inter-brain synchrony and memory were separately found to be related to
the emotional valence of events<sup>4–7</sup>. However, studies
connecting the dynamics of these three processes to commonly experienced
content are lacking. Traditional approaches have primarily relied on
research-grade equipment and restrictive experimental paradigms, with
limited research into the capabilities of consumer-grade EEG that are
accessible for use at home nor the element of privacy and personal
freedom that influences how participants experience TV shows at home
versus in a social, focus group setting. This gap suggested to us the
opportunity to develop an algorithm for populations in the real world
experiencing the same content.



Recent advancements in neuroimaging techniques provide for a detailed
exploration of neural synchrony, and the emergence of hyperscanning, the
neuroimaging of multiple individuals simultaneously, has proven the
utility of such algorithms for research and medical applications <sup>8,9</sup>. This progress has highlighted the usefulness of robust brain
synchrony algorithms and especially those that are suitable for
real-time analysis with consumer-grade neural activity measurement
devices that not only function effectively with lower signal quality and
fewer channels but additionally provide real-time feedback on neural
synchrony in naturalistic settings outside of the hyper controlled
laboratory<sup>10</sup>. The tantalizing possibility of measuring memory
formation effectively through these techniques led us to develop new
algorithms for EEG that would satisfy these criteria.



Correlated Components Analysis (CorrCA) has emerged as a promising
method for the study of inter-brain neural synchrony, enabling the
extraction of maximally correlated components across individuals' neural
recordings<sup>11</sup>. Projections from the hippocampus to the
prefrontal region, for example, we hypothesized to show up with CorrCA
while being missed by other methods. Indeed, CorrCA-derived
inter-subject correlations (ISC) were found repeatedly to be related to
the common cognitive aspects of experience across individuals in
response to stimuli, such as engagement <sup>11,12</sup> interest<sup>13</sup> and memorability<sup>3</sup>. CorrCA, as it was originally designed to
utilize EEG channels data as an input, allows the computation of scalp
projections, which provide valuable information about the spatial
distribution of correlated neural activity across the scalp <sup>11,14</sup>. Compared to the more familiar Canonical Correlation Analysis
(CCA), CorrCA is designed to find shared components across multiple
datasets or subjects while maximizing the correlation of these
components across the datasets. This approach also enables multiple
datasets as inputs from different subjects or experimental runs,
outputting a set of components that are maximally correlated across all
input datasets. A key assumption of corrCCA we accept is that there are
underlying components generated by the human brain that are shared
across datasets. In other words, that because the source of this data is
the human brain we accept as a sufficient common element of each data
source to validly map across participants by, while still allowing for
individual variations in the cognitive and emotional components meaning
of that data.



A benefit of methods in the corrCCA class is that we can interpret the
shared components across all datasets. In this study, we present an
enhanced approach to CorrCA, called features-based Correlated Components
Analysis (FCCA). This method examines the synchrony
level between individuals based on the dynamics of EEG activity features
rather than on denoised neural activity itself, thus focusing on
synchrony originating in the activity properties rather than physical
localization. That is the main contribution of this paper: FCCA by
orienting towards relationships in activity properties and not physical
space, is able to overcome spatial limitations of EEG to extract MRI
level insights for a fraction of the cost, and far more frequently since
the technique is possible from data recorded at home.



Our results demonstrate the validity of FCCA, and its potential to
derive insights from the dynamics of neural synchrony between subjects
about engagement, emotional positivity, and memory levels. The proposed
comprehensive algorithmic approach carries significance for contexts in
which there is a need for robust real-time estimation of group
experience in naturalistic settings, using consumer-grade recording
devices and, we believe, has far reaching implications for various
fields. Where memory is an important factor, such as education,
entertainment, and clinical applications, FCCA can inform strategies for
enhancing learning, creating memorable content, and advancing
understanding of emotional engagement dynamics and memory formation in a
variety of contexts.


<h2>Methods</h2>


We conducted two experiments using commercially available EEG headbands
with participants who watched the same set of video clips from home. The
first experiment involved short video clips viewing with subsequent
emotional ratings and a memory test. The second experiment included full
episode viewing followed by a memory test a few days later.


<h3>Participants</h3>


All participants had normal or corrected-to-normal vision, provided
written informed consent, and received payment for their participation.
Exclusion criteria included the use of medication that might influence
the experiment or the presence of neurological or psychiatric
conditions. Inclusion criteria required participants to have normal
vision or vision corrected to normal with contact lenses. In the first
experiment, there were 32 participants (mean age = 36, SD = 8.25, 16
females). In the second experiment, 27 participants (age 33, SD = 4
years, 13 females). For the memory test related to the American TV
episode in experiment 2, the subjects were distributed approximately
evenly across the five major regions of the continental United States
(Northeast, Southwest, West, Southeast, and Midwest) and all spoke
English as their native language.


<h3>Data Acquisition</h3>


In both experiments, participants viewed the videos within a mobile
Android app ("Arctop" by Arctop, Inc.), while their electrical brain
activity was recorded. EEG activity was recorded using Muse headbands
with EEG sensors embedded. For Experiment 1, the Muse 2 headband was
used, while the Muse S headband (Interaxon Inc.) was used for Experiment
2. Both devices are portable, noninvasive electroencephalography (EEG)
devices with a sampling rate of 256 Hz, containing four dry EEG sensors
located on the scalp at two frontal channels (AF7 and AF8) and two
temporal channels positioned behind the ears (TP9 and TP10), with a
reference channel at Fpz. The first experiment was conducted in an
office building with a support staff available to assist the participant
with the device and the experimental protocol, in the second experiment
each participant was mailed an Arctop technology kit at home that
included headphones (Sony), brain signal measuring headband, and a
tablet computer (Samsung) with the Arctop app installed. The headbands
were put on by the participants themselves, with the assistance of a
Quality Assurance (QA) screen that started before each session. The QA
showed the participants, in real-time, the data' quality, easily
directing them to adjust the headband properly for optimal signal
quality. In both experiments, recording was performed while the
participants were alone in a quiet room.




<Figure src="/deep-dives/pics/deep-dives/memory_decoding_figure1.webp" alt="Experiment 1: Short Clips. Experiment 2: Curb Your Enthusiasm Episode." caption={"Experiment 1: Short Clips. Experiment 2: Curb Your Enthusiasm Episode."} />




The experiment was composed of two sessions, separated by a break.
During these sessions each subject watched overall 38 short clips, each
ranging from 32 to 100 seconds in length, with a median duration of 59
seconds. The video clips collection was composed of clips with positive
(e.g. babies laughing), negative (e.g. An excerpt from "The Champ"<sup>15</sup>), and neutral affective valence (e.g. moving abstract shapes). In a
second experimental day, adapted from the study of Hasson et al.<sup>16</sup>, participants viewed a 27-minute-long episode from the
English-speaking television sitcom Curb Your Enthusiasm (Season 1,
Episode 7: "AAMCO" by Larry David). The episode depicted various
independent events, such as a dinner party and a minor car accident. A
list of the content in table.


<h4>Emotional Ranking</h4>


In the first experiment, after watching each clip, participants ranked
their emotional responses across eight emotions using a slider button in
the Arctop app: enjoyment, interest, happiness, dislike, boredom,
stress, relaxation, and sadness. We calculated the average score per
emotion for each stimulus.




<Figure src="/deep-dives/pics/deep-dives/memory_decoding_figure2.webp" alt="Experimental Design and Tasks." caption={"Experimental Design and Tasks."} />



<Figure src="/deep-dives/pics/deep-dives/memory_decoding_figure3.webp" alt="Memory test question examples for the video clips in experiment 1\n\t\t\tincluding recall and recollection." caption={"Memory test question examples for the video clips in experiment 1 including recall and recollection."} />



<Figure src="/deep-dives/pics/deep-dives/memory_decoding_figure4.webp" alt="Memory test question example for the long episode in experiment 2." caption={"Memory test question example for the long episode in experiment 2."} />


<h4>Memory Assessment</h4>


In both experiments, subjects weren't aware that they would perform a
memory test. Instead, after the end of the experiment they were notified
that there was an additional online task that they could participate in,
for which they would receive additional compensation. They received the
memory test via email 3 days after the video-watching session. We
verified that the email was received, and the participants were
instructed to perform the test in a quiet room.



In both experiments, the test was administered via a Google Form. Each
question included a still frame from the relevant time interval as a
visual cue and provided three alternative forced-choice answers.
Participants were instructed to choose one answer and rate their
confidence level on a Likert scale ranging from 1 to 6. Memory
performance was derived by calculating the percentage of correct answers
per question across subjects.



For the first experiment, the memory test was composed of 3 questions
about events from each of four of the clips (out of the 38 they
watched), i.e. 12 questions overall. For the second experiment, the
memory test consisted of 69 questions about the episode's narrative. The
questions were designed so that the events they addressed were separated
by approximately 20 seconds, and each question could be answered using
information from the relevant segment of the episode alone.


<h4>EEG-Based Between-Subjects Synchrony Estimation</h4>

<h5>Data Preprocessing</h5>


EEG data were preprocessed to remove artifacts and filtered to retain
relevant frequency bands. A band-pass filter (0.5–48 Hz) was applied to
each channel together with a notch filter (either 50 Hz or 60 Hz) to
remove line noise. During the performed tasks, 1.5 s of EEG data
segments were extracted from the filtered signal using a sliding window
with a stride of 100 ms. Data was excluded based on dropped packets of
EEG data length relative to the stimulus, which occurred due to
Bluetooth disconnections: deviation of more than a second was set as the
threshold for exclusion in both experiments. Besides excluding data
based on the recording duration threshold, recordings that had more than
5% missing values were excluded. Other data curation approaches were
taken to exclude invalid data while maintaining the sample sufficient
for each stimulus according to the heuristic of EEG-based CorrCA
calculations are valid for eleven participants and more<sup>13</sup>. Preprocessing was completed with feature extraction, where from
each EEG segment (epoch) a total of 54 features were extracted according
to the current version of Arctop software platform processing (July
2024), resulting in a matrix of epochs X features X subjects created per
stimulus to be used in subsequent FCCA analysis routines.


<h5>Correlated Components Analysis (CorrCA)</h5>


To examine the stimulus-evoked neural activity synchrony level between
subjects, we utilized CorrCA, which was applied to yield the ISC of the
neural responses <sup>3,12,13,13,17</sup>. Correlated component analysis
extracts projections of the data with maximal correlation by finding
linear combinations of the EEG data or features dynamics over time. As
described in detail by Cohen and Parra<sup>3</sup>, the procedure
involves the calculation of the pooled between-subject
cross-covariance,




<Figure src="/deep-dives/pics/deep-dives/memory_decoding_formula1.webp" />



and the pooled within-subject covariance,




<Figure src="/deep-dives/pics/deep-dives/memory_decoding_formula2.webp" />



where



<Figure src="/deep-dives/pics/deep-dives/memory_decoding_formula3.webp" />



measures the cross-covariance of all the time-series data sources in
subject <strong><em>k</em></strong> with all sources in subject <strong><em>l</em></strong>. When CorrCA is implemented on the EEG data, these sources refer to
the collected neural activity data, such as EEG-electrodes data. When
calculating it on the features data, instead of a matrix of time by
electrodes, the matrix is of time by features. Vector <strong><em>Xk(t)</em></strong> represents the scalp voltages, or features dynamics, at time <strong><em>t</em></strong> in subject <strong><em>k</em></strong>, and, <strong><em>x̄k</em></strong>, their mean value in time.



Obtaining the ISC involves simultaneously diagonalizing the pooled
covariance and the cross-covariance of the two data sets. The linear
components that achieve this can be obtained as the solutions of a
generalized eigenvalue equation:



<Figure src="/deep-dives/pics/deep-dives/memory_decoding_formula4.webp" />



Solving for



<Figure src="/deep-dives/pics/deep-dives/memory_decoding_formula5.webp" />



leads to ISC per component, as the component projections that capture
the largest correlation between subjects are the eigenvectors



<Figure src="/deep-dives/pics/deep-dives/memory_decoding_formula6.webp" />



of the matrix



<Figure src="/deep-dives/pics/deep-dives/memory_decoding_formula7.webp" />



with the strongest eigenvalues. Thus, the strength of correlation in the
ith component is obtained by:



<Figure src="/deep-dives/pics/deep-dives/memory_decoding_formula8.webp" />



High ISC is obtained when the dynamics of either EEG or features are
similar across subjects. Prior to computing eigenvectors, the pooled
within-subject correlation matrix is regularized to improve robustness
to outliers using shrinkage.


<h5>ISC Noise Level Estimation</h5>


To assess the ISC difference from noise level, i.e. remove
time-dependent intersubject entrainment while maintaining each subject's
temporal and spatial correlation, we implemented a phase scrambling
method <sup>18,19</sup>. This method involves selecting a random phase
offset for each subject and circularly shifting all data points along
the time dimension by this offset. Specifically, for each subject, we
first computed a feature matrix for the full EEG length. The circular
shift was then applied along the time axis of this matrix, followed by
segmenting the matrix according to event labels. The circular shift was
applied to each event separately and ISC was then calculated over the
circularly shifted data, either over the entire event (for overall ISC)
or by windows of 30 samples (for ISC dynamics). This process was
repeated one hundred times, with a new circular shift applied. ISC
computed in each iteration, yielding a distribution that estimated the
noise for each stimulus.


<h5>ISC Significance Test</h5>


The circular-shuffle-based distribution approach allows the generation
of a null distribution while maintaining the inherent structure of the
original data. The ISC p-value was computed per component. The p-value
of the overall ISC was calculated as 1 minus the cumulative distribution
of all circular-shift ISCs for a specific component. The dynamic p-value
was computed similarly for each time point.



To validate the approach of applying CorrCA on the dynamics of the
feature as opposed to the EEG signals, each ISC output of both
approaches was compared relative to the noise level. The p-value
relative to this distribution was calculated per stimulus, and ISCs
whose calculated p-value was smaller than 5% were considered
significantly different from noise. We estimated the fraction of
significant ISCs for the same set of stimuli, depending on the CorrCA
approach.


<h5>ISC Measures</h5>
<h6>Overall ISC</h6>



For the relation of the inter-brain synchrony with the emotional
ranking, we utilized the ISC for each stimulus, due to the rankings'
being referred to the entire stimulus. The ISC we refer to here is the
eigenvalue of the eigenvalue problem mentioned above. We extracted this
measure of the first three components.


<h6>ISC Scores</h6>


When looking into the relation of memory levels relative to the ISC
level of specific events or scenes in the videos, we calculated the ISC
dynamics, as it is expected different scenes evoke varying levels of
ISC. We compute the correlations in a time-resolved fashion<sup>11</sup>
by employing a sliding window with a 3-s duration with a shift of the window
occurring every second.



In the first experiment, we calculated the ISC scores of each of the
three first components in the time windows which correspond to the parts
of the video where the memory questions were asked, and their relation
to the memory accuracy. The ISC dynamics were averaged within the time
frame that corresponds to the event that the question refers to. The
mean ISC per event was normalized by subtracting the error rate of that
time frame, as calculated by the aforementioned noise estimation
procedure. Note that when the average ISC in the given time window is
higher than the error rate, the obtained value is positive, and vice
versa.



For the episode analysis, the ISC values were also calculated in
partially overlapping time windows as in the first experiment. Subjects
with missing values were removed from analysis at each time window. The
shuffle-based noise level per time window was subtracted from the ISC
value per window, to obtain a normalized ISC curve along the episode.
The normalized ISC curve within each memory test question's relevant
time frame was averaged, yielding an ISC measure per question. Both the
obtained ISC and the memory performance curves underwent convolution
with a moving average of 0.5 to 3 seconds running window.


<h5>Factor Analysis for Emotions Ranking</h5>


Factor analysis was performed to uncover latent structures within the
emotional rankings and their relation to ISC. As a preliminary step
towards conducting the factor analysis, we examined the
multicollinearity among the average emotional ranking by stimulus using
the Variance Inflation Factor (VIF), to ensure that the factors
extracted are not influenced by redundant or highly correlated
variables.


<h6>Variance Inflation Factor (VIF) Analysis</h6>


High multicollinearity, indicating redundancy, was identified for
emotions receiving scores between 5 to 10. We found that enjoyment and
happiness ranking exhibited moderate to high multicollinearity (VIF
values of 9.45 and 5.71, respectively). To address this issue, we
combined enjoyment and happiness into a single composite variable.
Following this step, the revised VIF values indicate that the issue of
multicollinearity has been effectively handled, with all variables
displaying low to moderate multicollinearity (minimum VIF of 1.68 and
Maximum VIF of 4.11) and, therefore, low redundancy.


<h6>Determining the Number of Factors</h6>


To determine the number of factors for the factor analysis, we employed
the scree plot method (named for the plot's resemblance to scree rock
cliffs in nature) and the Kaiser criterion (Supplementary Figure 1).
Based on this analysis, we proceeded with three factors for our factor
analysis. The resulting three factors in the factor analysis (after
inverting their signs for clarity) can be interpreted as "Emotional
Positivity," "Arousal," and "Engagement", respectively (Supplementary
Figure 2). Factor 1, "Emotional Positivity," is characterized by high
positive loadings for the enjoyment/happiness composite (0.581) and
relaxation (0.47), and high negative loadings for stress (-0.84),
dislike (-0.62), and sadness (-0.60). This factor represents a clear
contrast between positive and negative emotions. Factor 2, "Arousal,"
includes high positive loadings for dislike (0.48) and stress (0.43),
and high negative loadings for sadness (-0.74). This factor captures
emotions related to reactivity, stress, and sadness, and thus can be
interpreted as capturing arousal-related emotions. Notably, the second
component's ISC was found to be positively correlated with memory
accuracy. Factor 3, "Engagement," is defined by high positive loadings
for interest (0.77) and enjoyment/happiness composite (0.59), and high
negative loadings for boredom (-0.84). This factor represents engagement
versus disengagement.



These results further demonstrate the validity and efficacy of the FCCA
approach in yielding robust inter-subject correlations (ISC) as the
first three components revealed their distinct relationships with
cognitive processes, suggesting that ISC derived from FCCA is
informative about both emotion and memory processes in response to
content.


<h5>Relation of ISC to Memory Scores</h5>


In Experiment 1, we investigated the relationship between memory
accuracy for specific mini-scenes and ISC scores. ISC dynamics were
extracted for each stimulus, and the mean ISC within the time frames
corresponding to the mini-scenes was calculated and normalized by
subtracting the error rate. For each of the first three components, the
Shapiro-Wilk test was performed to choose between Pearson's and
Spearman's correlation coefficients. The p-values were adjusted using
the Bonferroni correction method. For Experiment 2, we assessed memory
accuracy and ISC dynamics using a range of convolution window sizes (0.5
to 3 seconds). The correlation between memory accuracy and ISC dynamics
was calculated, with the Shapiro-Wilk test performed to assess normality
and choose the appropriate correlation coefficient. Corresponding
p-values were adjusted for multiple comparisons using the Bonferroni
correction method.


<h4>Statistical Analysis</h4>


All statistical analyses were performed via Python (version 3.12.3;
"Statsmodels" and "SciPy" libraries). Pearson or Spearman correlation
coefficients were chosen based on the results of the Shapiro-Wilk test
for normality. The p-values that are reported refer to the hypothesis
test whose null hypothesis is that two samples do not correlate. All
p-values were adjusted for multiple comparisons using the Bonferroni
correction method.


<h2>Results</h2>


In the current study, frontal EEG activity was recorded using a Muse
headband while subjects watched video stimuli in two different designs
(Muse 2 for the first design and Muse S for the second). In an
offline-analysis, the data was preprocessed, EEG features were extracted
and neural synchrony was calculated. In the first design, each subject
watched 38 short clips (ranging from a minimum of 32 seconds to a
maximum of 100 seconds, a median of 59 seconds) and was asked to rank
eight emotions (enjoyment, interest, happiness, dislike, boredom,
stress, relaxation, and sadness; Figure 1a) after each clip. Three days
later, they performed a memory test to estimate their accuracy in
recalling mini-scenes from several clips viewed during the experiment
(Figure 1b). In the second design (Figure 1c), participants viewed a
27-minute-long episode and performed a memory test on mini-scenes from
the episode. In both designs, the subjects watched the clips and took
the memory tests alone. They were unaware that they would be taking a
memory test beforehand. See the designs' description elaboration in
methods.


<h3>Features-Based CorrCA Approach Validation</h3>


We calculated ISC for each stimulus based on the EEG features. The ISC
distributions of the first three components are plotted in Figure
2a. To assess their difference in the stimulus-specific noise
level, we created a "circular-shuffles"-based ISC distribution (i.e.
randomly shifting in time each subject's signals to remove
time-dependent intersubject entrainment). We compared the
discriminability from noise for ISC values derived from performing
CorrCA (i.e. calculated for the dynamics of the EEG signals) and FCCA
(i.e. calculated based on the EEG-derived features (see methods for
elaborated explanation). We found that the implementation of FCCA led to
ISC scores that were robust and were significant relative to shuffled
data (for 33 out of 38 stimuli; Figure 2b), to a greater extent relative
to EEG-based CorrCA (for 1 out of 38 stimuli; Figure 2c).




<Figure src="/deep-dives/pics/deep-dives/memory_decoding_figure5.webp" alt="FCCA Validation: distributions of ISC scores for the first three components across all stimuli." caption={"FCCA Validation: distributions of ISC scores for the first three components across all stimuli."} />




<Figure src="/deep-dives/pics/deep-dives/memory_decoding_figure6.webp" caption={"Arctop's novel FCCA method: ISC scores derived were significantly different relative to shuffled data for <strong>33 out of 38 stimuli, or 86.8%</strong>."} />



<Figure src="/deep-dives/pics/deep-dives/memory_decoding_figure7.webp" caption={"Current state-of-the-art CorrCA method: ISC scores derived from current state-of-the-art CorrCA were significantly different relative to shuffled data for <strong>1 out of 38 stimuli, or 2.6%</strong>."} />


<h3>
Mapping Emotional Space Relative to FCCA-Derived ISCs
</h3>


Having established that FCCA yields extremely robust ISCs that are
approximately 30 times more informative than the nearest methods, we
turned to examining the relationship between emotional responses and the
ISC values obtained by our approach. Multiple emotional responses for
each stimulus were ranked by each subject (see Figure 1a). Emotional
responses are inherently complex and multidimensional, involving a
variety of distinct but interrelated feelings (for example, feelings of
happiness and enjoyment might co-occur but not be fully redundant, as
might feelings of sadness and dislike).



Factor analysis enables us to reduce this complexity, and examine
whether the emotional rankings share common factors that are linked to
the ISC of each component separately. Due to its ability to uncover
latent structures that represent the combined effects of multiple
emotions, Factor analysis can potentially reveal underlying patterns
that may not be apparent in pairwise comparisons. Following eliminating
redundancy and determining the number of factors in a data-driven manner
(see methods), factor analysis was performed for 3 factors. The
resulting factors are interpreted as "Emotional Positivity," "Arousal,"
and "Engagement", respectively (Supplementary Figure 2).



<Figure src="/deep-dives/pics/deep-dives/memory_decoding_figure8.webp" alt="Factor-Analysis-Derived Factors Loadings For Emotional Ranking" caption={"Supplementary Figure 2: Factor-Analysis-Derived Factors Loadings For Emotional Ranking."} />




Visualization for the factor loadings of emotional rankings across three
factors (i.e. the contribution of each emotion to the identified
factors): "Emotional Positivity," "Arousal," and "Engagement." Each cell
represents the loading of an emotion on a specific factor, with the
color scale indicating the magnitude and direction of the loading.


<h4>Emotional Factors Correlations with ISC</h4>


To examine the relation between each of the emotional factors and the
ISC of the first three components, we performed a correlation analysis
and conducted Bonferroni correction for multiple comparisons. The
results indicate that only the "Engagement" factor has a significant
positive correlation with the ISC of the 1st component (r= 0.48,
p=0.006; Figure 3a), suggesting that a higher score for this component
is associated with higher engagement. "Emotional Positivity" and
"Arousal" did not show significant correlations with ISC of Component 1
(r=-0.16, r=0.05; Figure 3a). The ISC of component 2 was negatively
correlated with the "Emotional Positivity" factor (r=-0.44, p=0.015;
Figure 3b), but not to the other two factors. Interestingly, the ISC of
the third component was not found to be significantly correlated to any
of the factors (Figure 3c).




The "Engagement" factor showed a significant positive correlation with
ISC of the 1st component (r=0.48, **p=0.006). "Emotional Positivity" and
"Arousal" factors were not significantly correlated with ISC of
Component 1. The ISC of component 2 was negatively correlated with the
"Emotional Positivity" factor (r=-0.44, *p=0.015). ISC of the 3rd
component showed no significant correlations with any of the factors.



<Figure src="/deep-dives/pics/deep-dives/memory_decoding_figure9.webp" alt="Relation of Emotional Factors with ISC Components" caption={"Figure 3: Relation of Emotional Factors with ISC Components."} />


<h3>Memory Decoding from FCCA-Derived ISCs</h3>
<h4>
Short Clips
</h4>


We investigated the relationship between the memorability of short
scenes from a selection of short videos, and the ISC score (n=32, design
no. 1; Figure 5a). Specifically, we examined the ISC scores of each of
the three first components in the time windows corresponding to the
parts of the video where the memory questions were asked, and their
relation to the memory accuracy. To that end, we extracted the ISC
dynamics for each stimulus, calculated the mean ISC within the time
frame that corresponds to the mini-scene, and normalized it by
subtracting the error rate of that time frame (see methods; example in
Figure 4a). Note that when the average ISC in the given time window is
higher than the error rate, the obtained value is positive, and vice
versa.



A statistical analysis was performed to examine the correlation between
memory accuracy and the ISC scores. After applying Bonferroni
correction, we found that the only significant result was for the second
component's ISC, which had a positive Pearson correlation (r = 0.711, p
= 0.0284) between memory accuracy and the ISC scores. For the first
component, the analysis revealed a positive Spearman correlation (r =
0.663), however, this correlation was marginally significant (p = 0.06;
Supplementary Figure 3a). For the third component, the analysis showed a
weak Pearson correlation (r = 0.135, p = 0.676; Supplementary Figure 3b)
between memory accuracy and the ISC scores. This correlation was not
statistically significant before the Bonferroni correction.



Overall, these results suggest a specific relationship between the ISC
scores of the second component and the memory accuracy for the
corresponding video segments.




<Figure src="/deep-dives/pics/deep-dives/memory_decoding_figure10.webp" alt="Second Component's ISC Scores Relation to Memory Accuracy for Short Clips" caption={"Figure 4: Second Component's ISC Scores Relation to Memory Accuracy for Short Clips."} />




Example of ISC dynamics and the shuffle-based error rate, and the time
frame corresponding to the events the questions relate to. Shows
significant positive Pearson correlation between memory accuracy and ISC
scores for the second component (r=0.711, **p=0.0284).


<h4>Curb Your Enthusiasm Episode</h4>


We wanted to further examine the relation between memory decoding
capabilities and FCCA-generated ISC. First, we wanted to examine the
reproducibility of the specific relation to the second component's ISC
that we observed in the first experiment. Second, to gain a more
thorough perspective of the relation between the dynamics of these two
processes, by having dozens of mini-scenes within a longer stimulus, as
opposed to a diverse set where differences in brain states and latent
variables of attention and emotion may encode differently due to
juxtapositions in time alone. This approach allows us to characterize
the relation between fluctuations in the time series pair. Lastly, since
in the first experiment, the relation between the memory accuracy and
the first component's ISC was marginally significant, we sought to
further examine the nature of this relation.



To that end, we adopted a previously established fMRI experimental
procedure, which was designed to examine the relation between memory
accuracy and the level of inter-subject synchrony, along the time course
of a longer stimulus<a href="https://www.zotero.org/google-docs/?33CZRI"><sup>16</sup></a>. Specifically, we obtained memory accuracy for questions about 69
independent mini-scenes from a 35.5-minute "Curb Your Enthusiasm"
episode (n=11; design no. 2) and extracted the ISC scores that
corresponded to their time frames. This episode was chosen for its
preponderance of independent mini-scenes or events that are disjointed
from one another and thus do not necessarily contain information about
one or another, which allows examining memory accuracy for specific
time-points along the stimulus in a constructive way to relate it to the
ISC level.



The TV show stimulus was much longer, approximately a half hour, and
accordingly, the relevant events were spread over a duration of several
seconds. We accounted for the possible shift in time between the ISC and
the memory level time series in a data-driven manner by performing a
correlation analysis of the ISC score during the mini-scenes timing to
the accuracy in answering these questions, while smoothing both time
series with a moving average. Specifically, the correlation between the
memory level and the ISC score dynamics was calculated for a range of
running average convolution window sizes ranging from 0.5 seconds to 3
seconds, in increments of 0.5 seconds. A significance test was performed
to each convolution value and the corresponding p-values were adjusted
for multiple comparisons using the Bonferroni correction method.



We found that the relation between the memory accuracy and the second
component's ISC score dynamic was replicated in this design. The
correlation in the first convolution window size of 0.5 seconds was
found to be positive, however it was marginally significant, (r= 0.45,
p=0.09). After applying the Bonferroni correction, we found that
all other comparisons were statistically significant (for 1 second
p&lt;0.05, for 1.5 and above, p&lt;0.0001). This result further
corroborates the relation of the 2nd component's ISC and emphasizes its
stability. Figure 5a depicts the surprisingly stable and robust relation
between the ISC and memory accuracy of the 2.5-second convolution.




<Figure src="/deep-dives/pics/deep-dives/memory_decoding_figure11.webp" alt="Memory Accuracy and ISC Dynamics in the Curb Your Enthusiasm Episode" caption={"Figure 5: Memory Accuracy and ISC Dynamics in the \"Curb Your Enthusiasm\" Episode. Scatter plot of the ISC vs. memory accuracy as measured by multiple choice quiz answers showing that ISC score derived from FCCA is predictive of future episodic memory performance."} />




After corroborating the relation of memory accuracy to the 2nd
component's ISC, we turned to examine its relation to the 1st
component's ISC, as it was marginally significant in the first
experiment. We conducted the same set of analyses and statistical tests
and found no significant, or marginally significant results (Figure 5c).
This finding strengthens the specificity of the memory accuracy relation
of the FCCA-derived ISC of the second component.


<h2>Discussion</h2>


Our study introduces a novel approach to CorrCA, termed FCCA, which
examines synchrony based on EEG activity features. We found this method
to be highly robust, compatible with consumer-grade EEG devices, and
allows obtaining insights into the common emotional and cognitive
experiences based on the neural synchrony between subjects.



After establishing that on our data set, FCCA yields results that differ
from noise at a higher rate than CorrCA, we found that the FCCA approach
revealed distinct relationships between ISC of the first two components
and the video-clips' emotional ranking by the subjects. Our analysis
showed that the "Engagement" emotional factor strongly associated with
the first component's ISC. This result aligns with previous research on
the relationship between neural synchrony and engagement <sup>3,19,20</sup>, and therefore corroborates the validity of the FCCA approach.



Importantly, our study extends beyond engagement to explore the
relationship between neural synchrony, as measured by FCCA, and memory
which is known to be largely housed anotomically in deep structures that
are closed-fields to EEG signal. We found that the relation between ISC
and memory was specific to the second component. This result was evident
both when examining the memory for selected events taken from several
short video clips and also when these events were taken from the time
span of a longer episode. Our findings are in agreement with the
previous findings of Cohen and Parra<sup>3</sup>, who demonstrated the
relation between the first 3 components' ISC relation to memory level
when applying the CorrCA analysis to EEG data. Therefore, the ISC
dynamics derived by our algorithm can effectively capture memorability
level dynamics while individuals experience the same stimulus.



The second component's overall ISC and the "Emotional Positivity" factor
were negatively correlated. This finding aligns with the study of
Nummenmaa et al.<sup>21</sup>, reporting an association between negative
valence and increased inter-brain synchrony in the emotion-processing
network, including prefrontal regions. Together with our result that the
second component's ISC level corresponds to the memorability dynamics
along the stimulus, our findings point to an inverse relationship
between emotional positivity and memory accuracy. This finding is in
agreement with evidence suggesting that sad or negative stimuli are more
memorable<sup>6</sup>. Future research could explore how the dynamics of
emotional valence along the stimulus correspond to the ISC dynamics, and
how different emotional valences in context interact with memory
formation.



As FCCA provides a comprehensive framework that is suitable for either
offline or real-time tracking of emotional and cognitive processes
simultaneously, future studies could also examine how different types of
stimuli, ranging from educational materials to various forms of
entertainment, affect experience. This approach may facilitate creation
of personalized content in education and entertainment, potentially
informing the design of engaging and memorable content. Finally,
future studies could extend the use of this robust algorithm to the
hyperscanning research field, by enabling tracking emotions, and memory
processes based on large groups neural dynamics.



Previous findings that relate the synchrony level calculated by CorrCA
to individual traits, such as age<sup>22</sup> and familiarity or expertise
with respect to the displayed visual stimuli<sup>23</sup> (Goldberg, 2014).
Therefore, given the potential applicability of FCCA for naturalistic settings,
it may be implemented for identifying divergence from a common synchronized
neural response, allowing the detection of abnormal patterns in individuals'
traits, such as emotional reactivity or cognitive processing. This avenue
may be highly relevant in educational and clinical settings, in the
diagnosis and treatment of disorders affecting social cognition or memory.



In conclusion, our study demonstrates that our proposed approach is a
powerful and versatile tool for investigating neural synchrony. By
providing a nuanced understanding of how emotional responses and memory
processes are reflected in neural synchrony, FCCA opens new pathways for
studying common patterns within and between collective experiences
across various contexts. The ability of FCCA to yield meaningful results
across different kinds of consumer-grade recording devices significantly
enhances its generalizability, expanding the scope of real-time tracking
of neural synchrony in naturalistic settings that is possible through a
variety of headwear from earbuds with temporally located EEG sensors to
AR/VR devices with frontally located EEG sensors.


<h2>Supplementary</h2>


<strong>E</strong>igenvalues of the principal components in descending
order for emotional factors. The Kaiser criterion, which retains factors
with eigenvalues greater than 1, supported retaining three factors.




<Figure src="/deep-dives/pics/deep-dives/memory_decoding_figure12.webp" alt="Supplementary Figure 1: Scree plot for choosing the number of factors for Factor Analysis." caption={"Supplementary Figure 1: Scree plot for choosing the number of factors for Factor Analysis."} />


<h2>References</h2>


1. Nummenmaa, L., Lahnakoski, J. M. &amp; Glerean, E. Sharing the social
world via intersubject neural synchronization. <em>Curr. Opin. Psychol.
</em><strong>24</strong>, 7–14 (2018).



2. Hasson, U. &amp; Frith, C. D. Mirroring and beyond: coupled dynamics
as a generalized framework for modeling social interactions. <em>Philos. Trans. R. Soc. B Biol. Sci.
</em><strong>371</strong>, 20150366 (2016).



3. Cohen, S. S. &amp; Parra, L. C. Memorable Audiovisual Narratives
Synchronize Sensory and Supramodal Neural Responses. <em>eNeuro
</em><strong>3</strong>, (2016).



4. Kensinger, E. A. Remembering the Details: Effects of Emotion. <em>Emot. Rev.
</em><strong>1</strong>, 99–113 (2009).



5. Santamaria, L. <em>et al.</em> Emotional valence modulates the topology
of the parent-infant inter-brain network. <em>NeuroImage </em><strong>207</strong>, 116341 (2020).



6. Cunningham, T. J., Leal, S. L., Yassa, M. A. &amp; Payne, J. D.
Post-encoding stress enhances mnemonic discrimination of negative
stimuli. <em>Learn. Mem. </em><strong>25</strong>, 611–619 (2018).



7. McGaugh, J. L. The amygdala modulates the consolidation of memories
of emotionally arousing experiences. <em>Annu. Rev. Neurosci.
</em><strong>27</strong>, 1–28 (2004).



8. Czeszumski, A. <em>et al.</em> Hyperscanning: A Valid Method to Study
Neural Inter-brain Underpinnings of Social Interaction. <em>Front. Hum. Neurosci.
</em><strong>14</strong>, (2020).



9. Dumas, G., Nadel, J., Soussignan, R., Martinerie, J. &amp; Garnero,
L. Inter-Brain Synchronization during Social Interaction. <em>PLoS ONE
</em><strong>5</strong>, e12166 (2010).



10. Nazneen, T. <em>et al.</em> Recent Trends in Non-invasive Neural Recording
Based Brain-to-Brain Synchrony Analysis on Multidisciplinary Human Interactions
for Understanding Brain Dynamics: A Systematic Review. <em>Front. Comput. Neurosci.
</em><strong>16</strong>, 875282 (2022).



11. Dmochowski, J. P., Sajda, P., Dias, J. &amp; Parra, L. C. Correlated
Components of Ongoing EEG Point to Emotionally Laden Attention – A
Possible Marker of Engagement? <em>Front. Hum. Neurosci. </em><strong>6</strong>, (2012).



12. Poulsen, A. T., Kamronn, S., Dmochowski, J., Parra, L. C. &amp;
Hansen, L. K. EEG in the classroom: Synchronized neural recordings
during video presentation. <em>Sci. Rep. </em><strong>7</strong>, 43916
(2017).



13. Dmochowski, J. P. <em>et al.</em> Audience preferences are predicted
by temporal reliability of neural processing. <em>Nat. Commun.
</em><strong>5</strong>, 4567 (2014).



14. Truong, N. C. D. <em>et al.</em> Mapping and understanding of correlated
electroencephalogram (EEG) responses to the newsvendor problem. <em>Sci. Rep.
</em><strong>12</strong>, 13800 (2022).



15. Gross, J. J. &amp; Levenson, R. W. Emotion elicitation using films. <em>Cogn. Emot.
</em><strong>9</strong>, 87–108 (1995).



16. Hasson, U., Furman, O., Clark, D., Dudai, Y. &amp; Davachi, L.
Enhanced Intersubject Correlations during Movie Viewing Correlate with
Successful Episodic Encoding. <em>Neuron </em><strong>57</strong>,
452–462 (2008).



17. Cohen, S. S. <em>et al.</em> Neural engagement with online educational
videos predicts learning performance for individual students. <em>Neurobiol. Learn. Mem.
</em><strong>155</strong>, 60–64 (2018).



18. Prichard, D. &amp; Theiler, J. Generating surrogate data for time
series with several simultaneously measured variables. <em>Phys. Rev. Lett.
</em><strong>73</strong>, 951–954 (1994).



19. Parra, L. C., Haufe, S. &amp; Dmochowski, J. P. Correlated
Components Analysis - Extracting Reliable Dimensions in Multivariate
Data. <em>Neurons Behav. Data Anal. Theory </em><strong>2</strong>,
(2019).



20. Dikker, S. <em>et al.</em> Brain-to-Brain Synchrony Tracks Real-World
Dynamic Group Interactions in the Classroom. <em>Curr. Biol.
</em><strong>27</strong>, 1375–1380 (2017).



21. Nummenmaa, L. <em>et al.</em> Emotions promote social interaction by
synchronizing brain activity across individuals. <em>Proc. Natl. Acad. Sci.
</em><strong>109</strong>, 9599–9604 (2012).



22. Petroni, A. <em>et al.</em> The Variability of Neural Responses to Naturalistic
Videos Change with Age and Sex. <em>eNeuro </em><strong>5</strong>,
ENEURO.0244-17.2017 (2018).



23. Madsen, J., Margulis, E. H., Simchy-Gross, R. &amp; Parra, L. C.
Music synchronizes brainwaves across listeners with strong effects of
repetition, familiarity and training. <em>Sci. Rep. </em><strong>9</strong>, 3576 (2019).



24. Goldberg, H., Preminger, S., &amp; Malach, R. (2014). The
emotion-action link? Naturalistic emotional stimuli preferentially
activate the human dorsal visual stream. Neuroimage, 84, 254-264.



25. Lee, H., Chen, J. Predicting memory from the network structure of
naturalistic events. Nat Commun 13, 4235 (2022).



26. Rugg, M. D. &amp; Curran, T. Event-related potentials and
recognition memory. Trends Cogn. Sci. 11, 251–257 (2007).



27. Rugg, M. D. &amp; Vilberg, K. L. Brain networks underlying episodic
memory retrieval. Curr. Opin. Neurobiol. 23, 255–260 (2013).



28. Simony, E. et al. Dynamic reconfiguration of the default mode
network during narrative comprehension. Nat. Commun. 7, 12141 (2016).



29. Goldberg, I. I., Harel, M. &amp; Malach, R. When the brain loses its
self: prefrontal inactivation during sensorimotor processing. Neuron 50,
329–339 (2006).



30. Summerfield, C. et al. Neocortical connectivity during episodic
memory formation. PLoS Biol. 4, e128 (2006).

---

# Five Levels of Explanation (Part II): How Brain-Computer Interfaces Work
URL: https://arctop.com/deep-dives/how-bci-works-part-2
Published: 2023-12-29

This is a continuation of our previous post where we explain brain-computer interfaces in five levels of difficulty, following <a href="https://www.youtube.com/results?search_query=wired+five+levels" target="_blank">WIRED's</a> style. We recommend you first read <a href="https://arctop.com/deep-dives/how-bci-works-part-1">Part 1</a> for levels 1 through 3. Now let's dive into levels 4 and 5.&nbsp;



<h2 id="level-4">Level 4: Grad Student</h2>



With the previous background knowledge, as a 'graduate student' for the purpose of this blog post's structure, you might wonder what the key research questions in the BCI field are. What are challenges that are valuable and may be solvable in the next 3-5 years? In other words, what could be worthwhile questions to work on for a Masters of Ph.D.&nbsp;All three pillars of BCI, hardware, algorithms and applications, have ripe questions here at the close of 2023.





Below we go more into some of these below, in a word these are <em>hardware</em> challenges, which abound in the size, weight, power, cost of BCI devices; <em>software</em> challenges where the overarching mission of understanding the brain is paramount (and possibly an impossible quest), and in <em>applications</em> the challenge of effectively achieving the user's goal for use, i.e. quality, is all that matters. All in all combine to create the value of BCI.



<h3>
Hardware: SWaP-C Challenges to Mainstream BCI
</h3>



Historically, computer technology has gotten smaller and smaller. The almighty 'miniaturization' principle. Though it may not sound profound, it is, and this trend is one of the reasons why BCI technology is now becoming available to people for everyday use. Electronic technology keeps getting smaller and smaller mainly because smaller devices fit into our lifestyles better. The computer industry is now regularly etching precise, detailed designs on the order of nanometers into silicon chips so our phones and computers can be lighter while also being more powerful: to emphasize — nanometers with precision — a supremely amazing feat.





It's fair to say that as a species we have been fairly obsessed with miniaturization of our technology and very successful in scratching that itch. An extraordinary amount of engineering imagination and efforts have gone into shrinking components and making the already small still smaller, and the trend continues. The computer scientist Gordon Bell found that miniaturization is actually happening in regular intervals over time — generating new classes of computers in the process that, <a href="https://computerhistory.org/blog/the-worlds-smallest-computer/" target="_blank">in his words</a>:





“…bring with them new markets, ecosystems, and — most importantly — new types of users.”





<Figure src="/deep-dives/pics/deep-dives/five_levels_part2_figure1.webp" alt="The evolution of computers" caption={"The evolution of computers where smaller sizes leads to new classes of products. This progression continues today into the 2020s as BCI is becoming a part of consumer technology. Courtesy: Computer History Museum."} />





There is a natural combining that is currently underway with BCI
technology being embedded in headphones, earbuds and AR/VR form factors
converging to the human head and globally opening the era of BCI in
consumer electronics. The electronics for amplification, filtering,
digitization, communication underpinning these new capabilities have
been consistently miniaturized, but brain sensors have been more
constrained due to the biophysical properties of conductive ionic
currents on the skin where sensor surface area is correlated with
signal-to-noise-ratio (SNR). As sensors become integrated with existing
technology and head-wear form factors the size won't matter much since
it will be small enough to be functional invisible to the user.&nbsp;





Devices that are barely visible to the human eye already are designed to
fit seamlessly into, and around, our bodies; the progression trends
straightly towards the nanoscale. And already, these implanted sensors
have allowed humans and computers to interact with each other directly.
These new types of symbiotic relationships transform not only daily
lives, but our entire species in the process; it's as if one day we will
all grow another limb or develop, theoretically, a new sense. According
to Bell, we are approaching the apex of small electronics. With that
convergence to the minimum scale for electronics there may then be
inversion: with technology going inward and contorting itself in new
dimensions. As in, into our bodies — implants that seamlessly, for
example, measure from the brain with a Young's Modulus matching jiggle,
so the implant does not shear and damage tissue as it naturally jiggles
about in daily life.





The key research challenge in this pillar is how to make BCI sensors and
devices easy to use, comfortable, affordable, unobtrusive, and reliable
for long-term monitoring in daily life. In the past one to two decades,
there have been efforts in both academic and industry and significant
advancement in both invasive and non-invasive BCI technology. On the
research side, continual breakthroughs have been made in materials of
sensors that are biocompatible, flexible, and can form stable contact
with skin - in the form of tattoos or fabrics for example. Another
important direction is how to miniaturize the sensors and devices such
that they are easy to put on by the users themselves, comfortable to
wear, and can last for a long time, all the while recording reliable
signals from the brain with sufficient spatial coverage.





<Figure src="/deep-dives/pics/deep-dives/five_levels_part2_figure2.webp" alt="Skin-integrated 'electronic tatoo' sensors" caption={"Skin-integrated 'electronic tatoo' sensors can measure things like skin hydration, temperature and electric signals from brain and muscle activity. Images courtesy of <a href=\"https://rogersgroup.northwestern.edu/\" target=\"_blank\">Rogers Research Group.</a>"} />





We now see more companies tackling this particular challenge, delivering
BCI products in the form of earbuds, headphones, eyewear, or that
integrate with a VR headset or a hearing aid. Lastly, it's important to
point out the breakthrough in invasive technology. Research labs and
companies are pushing the boundary that maximizes the number of sensors
and their spatial coverage of implanted devices while minimizing the
risk and costs of surgical operations and long-term use.





These are highly active research and development areas that will likely
have many breakthroughs in the next 5 years. One framework for thinking
about where to focus as a graduate student in a hardware field is the
acronym: SWaP-Cs. Which stands for Size, Weight, Power, Cost — the goal
is to decrease all of these variables. There's a dual meaning to it
also: Style, Washability, Prestige, Comfort. Perhaps the areas that will
advance BCI most today are in these alternative SWaP Cs where sexy, cool
and comfortable meet in headwear products like glasses or earrings,
hats, beanies and bindis, all supreme expressions of what a BCI can be.
What will BCI devices look like in the future? Inspired by Mark Weiser’s
view, we believe BCI will disappear into the fabric of everyday life and
be indistinguishable from it. At Arctop we think of the technology as
being an extension of cognition – of our everyday thinking, expanded and
extended, embodied in the environment and technologies around it -
without being felt as a technology. That's how seamless it should
be.&nbsp;





Without the capability of remote sensing at a distance away from the
brain activities, BCI sensors will need to be around, on, or inside the
head. With such constraints, there is limited real estate of the head
where people can (and are willing to) wear a device for a long time.
This is why we are already seeing and will see even more in the near
future, that BCI technology being miniaturized and integrated into
existing eyewear or headwear, including eyeglasses, headphones, earbuds,
hearing aids, headbands, helmets, and VR / AR headsets is the
trend.&nbsp;





<Figure src="/deep-dives/pics/deep-dives/five_levels_part2_feature.webp" alt="Arctop compatible BCI" caption={"Two example BCI form factors that have recently become possible thanks to hardware advances (<a href=\"https://arctop.com/deep-dives/hardware\" target=\"_blank\">link</a>)."} />





Another trend is invasive technology - making implants smaller, safer,
and more affordable. For now, the risk of invasive technology may only
be justified by clinical or medical use for those in need, and the
technology development still needs to go through rigorous clinical
validation and regulatory processes to de-risk the adverse effects in
long-term use. But in the long run, when it reaches a similar risk level
as replacing a missing tooth, in exchange for unlocking greater human
abilities, would you be willing to do so? We might be.





<Figure src="/deep-dives/pics/deep-dives/five_levels_part2_figure3.webp" alt="BCI system within a VR environment" caption={"Schematic showing a BCI system within a VR environment. User interactions rapidly customize virtual objects and content is tailore to users interactively in a closed loop that continuously improves the BCI for that individual based on reinforcement learning. Parallel reinforcement learning loops serve to optimize the entire environment and all user interactions. <a href=\"https://patents.google.com/patent/US11662816B2/en\" target=\"_blank\">Interactive electronic content delivery in coordination with rapid decoding of brain activity</a>. (Furman, Kwalwasser, 2023)."} />



<h3>
<strong>Software: Algorithms for Decoding Brain Data and Infrastructure</strong>
</h3>



<Figure src="/deep-dives/pics/deep-dives/five_levels_part2_figure4.webp" alt="Empathic Computing System and Methods for Improved Human Interactions With Digital Content Experiences" caption={"<a href=\"https://patents.google.com/patent/US20210390366A1/en\" target=\"_blank\">Empathic Computing System and Methods for Improved Human Interactions With Digital Content Experiences.</a> Information hierarchy showing how multimodal data and priors feed into an application environment. As population data grows, algorithm capabilities will too. (Furman, Kwalwasser, 2021)"} />





The decoding algorithms of BCI technology have mainly used digital
signal processing and statistical machine learning techniques.
Artificial neural networks have not worked so great traditionally for
real neural network activity, but with recent breakthroughs in
artificial intelligence (specifically deep learning methods) for text,
image, and speech recognition, there has been an outpour of <a href="https://iopscience.iop.org/article/10.1088/1741-2552/ab0ab5/meta" target="_blank">publications</a> applying deep-learning methods to biosignals. But deep-learning has not
been a magic wand that magically solves the recognition problem for brain
activities yet. Why? The key challenges lay in both the unique data and algorithms.





On the data side, unlike text and images, brain data has limited numbers
of public, large-scale datasets for training models and benchmarking
performance on. These datasets often have their own data acquisition
hardware and idiosyncratic data collection protocols which introduce
more variability. There is also a lack of standards on metrics and
criteria for controlling signal quality and consensus for data cleaning
methods despite wide consensus on the need for cleaning the always noisy
signal. Lastly, limited "ground truth" labels are available and they
often have low temporal resolution with high variability (e.g.
disagreement even between “experts”).





But in the last decade, significant efforts have gone into addressing
these challenges and we are optimistic that these are likely solvable in
the near future. For example, some BCI research communities and
non-profit organizations have started to publish their datasets (<a href="https://openneuro.org/" target="_blank">OpenNeuro</a>) and open-source codebase (<a href="https://github.com/NeuroTechX/awesome-bci#Software" target="_blank">NeuroTechX</a>) to establish and follow a standard for data and label formats, for
example 'EEG-BIDS' (<strong>Pernet et al 2019</strong>).&nbsp;Effective
methods for handling artifacts and noises in EEG in the real-world
setting have been proposed and evaluated (<strong>Chang et al 2019</strong>), and additional data modalities are being used (e.g. behaviors from
phones, camera, physiological signals) to provide rich context and
automated labels for the EEG. With these advances, we may be able to
reach a critical point with enough data and labels that can drive the
breakthrough in algorithm development.





On the algorithm side, for readers who are AI researchers and developers
wanting to transfer AI breakthroughs from other domains and solve the
brain’s grand challenge, there are a few unique challenges for brain
signals. First is the “context” or labels during which brain signals are
collected. Unlike an image of a dog where the ground truth is
unambiguous and universal, the ground truth for a period of brain signal
can be noisy and subjective. There is not, for example, consensus about
how many emotions humans can feel or the best way to establish the
timing of when they are being felt. Hence approaches like
self-supervised learning and multi-modal data for automated label
generation are rising and necessary.&nbsp;





The second challenge is that the brain activities are always changing.
Coming from sources throughout the brain, that overlap and transient
signals that propagate in complex ways, there is a lot to analyze.
Interestingly, brain activities seem to transition from a stationary
state to another, nonstationary state at unbound timescales, like
progression of sleep stages or a sequence of thoughts. But these brain
states do not have fixed time intervals. Learning the “unit” of brain
states, like learning the “vocabulary” of speech, may be the key to
reduce the complexity of the brain decoding challenge and relax the data
requirements. By transforming the problem from decoding time-series data
to sequence data (e.g. leverage methods like brain-state modeling, 
<strong>Hsu et al 2018, 2022</strong> or <a href="https://ai.meta.com/blog/advancing-direct-speech-to-speech-modeling-with-discrete-units/" target="_blank">speech-to-unit translation</a>) 
dimensions can be reduced and patterns can be modeled more
effectively from first principles, obviating the need for massive data
sets thay require more parsing.





The third challenge is the “human factor” that changes the BCI
performance from day to day or people-to-people due to differences in
brain anatomy, sensor locations, or users’ states like attention,
emotion, and motivation (<strong>Lotte et al 2013</strong>). To tackle
this challenge, transfer learning techniques are needed to use data and
pre-trained models from other days, other users, or even other devices
to facilitate the “re-calibration” of your model (<strong>Chiang et al 2021</strong>). Adaptive learning is also being used to automatically, and
continuously adapt the model to the user.&nbsp;



<h3>
Applications
</h3>



<a href="https://bnci-horizon-2020.eu/" target="_blank">BNCI Horizon 2020</a> (<strong>Brunner et al 2015</strong>), one of the highly influential BCI projects, presented five
categories of BCI use cases: replace, restore, improve, enhance, and
research. Here we propose a mental framework to understand the utility
of BCI applications and to inspire fellow and future BCI pioneers to
build upon.





<Figure src="/deep-dives/pics/deep-dives/five_levels_part2_figure5.webp" alt="BCI Applications" caption={"Map of BCI Applications (proposed by the authors). The horizontal axis represents, from left to right, restoring/replacing existing abilities, improving current abilities, and unlocking new abilities. The vertical axis represents body functions (movement and senses) versus brain functions (cognition and intention)."} />





BCI applications started off from “replacing body ability” (bottom left
in the Utility Map) for paralyzed patients. With advancements in
invasive technology, particularly implants and decoding capability, we
will see clinical and medical-use BCI achieve higher accuracy,
reliability, and degree of freedom of control for more diverse
populations with disabilities. This includes finer movement control and
speech decoding that would significantly increase the communication
bandwidth and speed.





BCI then expands toward “restoring brain ability” (upper left in the Map
of BCI&nbsp;Applications) for patients with neurological disorders by
coupling the brain wave to provide real-time, closed-loop stimulation
for therapeutic interventions. BCI coupled with electrical or magnetic
stimulation techniques, either non-invasively (TMS, tACS, tDCS, taVNS)
or invasively (DBS, VNS or FES), have already been used or entered into
clinical trials for treating a variety of neurological or mental
disorders. Auditory stimulation in sync with slow-wave brain activities
in sleep may improve sleep and potentially prevent cognitive decline (<strong>Zeller et al 2023</strong>). Visual feedback (e.g. play/stop a video clip) can be provided to
incentivize users to achieve an ideal brain state for improving
cognitive functions in people with ADHD or Schizophrenia (<strong>Singh et al 2020</strong>).





Now we are seeing the trend in development and applications toward
improving current abilities and unlocking new abilities (bottom and
upper right in the Utility Map). Passive-BCI applications (<strong>Zander and Kothe, 2011</strong>) have been used in day-to-day settings for healthy populations in
scenarios like learning in a classroom, working in the office or at
home, driving a car, navigating an airplane, and playing and integrating
in virtual-reality games. We will explore this topic further in Level
5.





To sum up, with increasingly comfortable and miniaturized headwear for
brain-sensing, the accumulation of data and labels, breakthroughs in
artificial intelligence systems and computing capability for decoding,
we will see BCI technology flourish and provide a better quality of life
for&nbsp;us.



<h2>Level 5: Expert</h2>



What is the trend in BCI technology in the next 5-10 years and
beyond? What are some grand challenges in BCI that may require
breakthroughs from other fields? In this section, we invite you, as an
expert in your own field, to collaborate and help unlock the future of
BCI.



<h3>
Scientific Understanding of the Brain
</h3>



"Complete understanding of the human brain" is the grand challenge. It
is so grand that almost certainly we will not arrive at a satisfactory
result in this lifetime, one ought to walk humbly when approaching this
mountain with its peak disappearing into the clouds. Like an ant
learning the latest mathematical model of how ant societies work, our
understanding may ultimately be limited by our vocabulary no matter how
fluent we become.





The goal can be pursued incrementally and eventually the mountain of
knowledge needed might be scaled. We will know we've arrived when all
neurological diseases, neurodegenerative disorders, mental illness and
anything negative related to the brain does not exist anymore– since if
we have complete understanding of it, it will mean we have complete
ability to modify it. Before getting lost out on a philosophical or
bioethics branch here lets opine back on the side of neuroscience and
the grand challenge of the field: understanding the brain to the best of
our abilities.





The phenomena observed and documented by neuroscientific instruments
seem only scratching the surface. Particularly, three research areas
might have a direct impact on the future of BCI:





(1) understanding the mechanism that gives rise to the complex and
dynamic cognitive functions and mental states in humans. What are the
neural mechanisms and cognitive processes that embody our subjective
experience of feelings, thoughts, or other mental states? Without the
bridges between neuroscience, cognitive science, and psychology, we
won't be able to quantitatively measure the various mental states. How
many emotions do we have after all? To measure them all we need to know.





(2) advancement in tools to record and stimulate the brain with high
spatial and temporal resolution - this will not only advance our
scientific understanding of the brain but also provide BCI developers
with the right tools to decode and modulate the desired brain functions.
New tools lead to new rules, as the saying goes. And circularly, new
rules to new tools.





(3) using BCIs as a research tool itself, serving as a synthetic nervous
system, that helps advance our understanding of neuronal, functional,
and psychological changes in response to real-time, dynamical, adaptive
feedback. Such understanding can create wondrous values, for example, to
treat neurological or psychiatric disorders, find personalized methods
to optimize our learning, promote our brain and mental health, and
effectively co-evolve with the rapidly advancing artificial intelligent
personal agents.



<h3>
BCI Software Platform: from Algorithms to Mechanisms
</h3>



Naturalistic interactions with computer applications is one primary goal
of BCIs. Many of the most popular BCI paradigms focused on decoding
algorithms and artificially connected the outputs for other forms of
control as a proof-of-concept demonstration. These often are not natural
for users and the interactions they create with applications can be
unintuitive: flickering lights or flashing bars on a screen lead to a
word being selected, or imagining opening or closing a hand steers a
wheelchair. Many of these paradigms overlook the human in the loop and
principles of usability that other personal computing areas.





The connection between decoding algorithms and forms of interactive
feedback or control, which together we refer to as the “Mechanism”,
requires thoughtful and user-centered design. For a BCI to be useful and
used on a daily basis, it needs to be comfortable to the extent of
becoming unnoticeable. Not only physically, but psychologically as well
in terms of the methods it employs to interact with the user. Here we'll
offer a few unique BCI control and interaction mechanisms.





<Figure src="/deep-dives/pics/deep-dives/five_levels_part2_figure6.webp" alt="BCI System Diagram" caption={"Types of BCI Control & Interaction mechanisms (proposed by the authors)."} />





<strong>Actions</strong> mechanism can be produced by a user actively doing
something and expecting fast and accurate outcome, like silent speech for
communication or imagery movement for object control. <strong>Intent</strong> mechanism is a more subtle and natural form of control where a user wants
something to happen by naturally changing what they focus on, prefer over,
or dislike. <strong>States </strong>mechanism does not require any
mental efforts and is a more subliminal, passive measure that reflects
user emotions and general mental conditions. <strong>Responses</strong> mechanism
also does not require active control but measures a user’s natural mental
reaction to external stimuli or feedback, like seeing and correcting a mistake.





Traditional human-computer interfaces only provide Actions mechanisms,
such as pressing an icon with one's finger to open an app. It requires
active effort to move the finger and press, and one would expect the app
to open instantly every time it is pressed. Traditional BCI has followed
this broader computing trend and pursued Action mechanisms. However, BCI
has the unique ability to unlock other types of control and interaction
mechanisms like Intent, States, and Responses in addition. These
mechanisms require less mental efforts from users, and because of the
more implicit control mechanisms, users do not have the same high
expectation on performance compared to Actions. A successful BCI will in
this way expand the command repertoire to provide various mechanisms
that are intuitive to users, similar to how touchscreens and computer
mice were instantly familiar and easy to start using by most people
right away when they came out.&nbsp;





The ultimate solution likely involves a suite of mechanisms that
inter-relate and interact with each other while driving external
actions. Much like the brain itself is arranged, the aspects of
cognition that connect to applications through a BCI likely need to be
hierarchically arranged and segmented to work best.



<h3>
What are the Killer Applications of BCI?
</h3>

<h4><em>1. New Communications</em></h4>



Full conversational speech and full control over a computer are the
goals for personal electronics generally when it comes to
communications. Hands-free, eyes-free, voice-free, gesture-free commands
that will work for patients and physically-able people alike. Pure
intention and intuitive interactions easily working every time.





In the first 50 years of BCI research, most applications focused on
active control within medical, clinical, and research ecosystems to
allow people who are paralyzed a new way to communicate or control a
device that increases their independence. The noble goals and efforts in
this arena have been mostly for helping patient populations where 80%
control over a cursor to someone who is locked-in is immediately life
changing. Continuing along this line of development, it is a feasible
goal of the field to free every single locked-in patient and that, we
believe, is achievable in the next decade with noninvasive methods. If a
person has functional brain activity, they should be able to communicate
basic commands at the least. This is a humanitarian objective of the
field and needs to be achieved. Making locked-in a thing of the past
remains a powerful driver for many in the field. For some it is the only
litmus test worth evaluating progress by.





From another perspective though, for most people, a cursor that only
works 80% of the time is simply not good enough and the dominant BCI
paradigms of the existing research feel strained and uncomfortable to
use since they were designed for the most desperate of cases. In the
past decade, a new ecosystem has been emerging towards a more natural
use of BCIs where there is no need for active control. This demand on
the user is lifted and interaction is more seamless. Easier connection
paradigms makes BCI more accessible to average consumers. While
implanted BCI systems might be needed to communicated at conversational
speeds, there has been success in noninvasive systems that is
accelerating, with adjacent advances in generative AI helping to power
the progress.



<h4><em>2. Skills Learning</em></h4>



There are already killer applications within learning, mostly tied to
how fast the same skills can be taught with a BCI and without. For
example, when you are learning in an online course, an add-on BCI can
accelerate your learning by sensing that you are bored and giving you
harder materials; or sense that you are overwhelmed and slow down with
more examples; tracking to optimal performance for people individually
according to Yerkes-Dodson-like performance curves (<strong>Yerkes, Dodson, 1908</strong>).





A driving trainer with add-on BCI can sense you feel stressed and modify
its driving behavior to help you feel comfortable and in a more extreme
example, an airline pilot can train in difficult and dangerous
conditions that simulate their state of mind more effectively. Or
imagine having a personal athletic trainer who can give you prompt
notice when you seem to be distracted - to regain control of your
attention at a key moment of an exercise. The keyword here is "action" -
BCIs can take actions and turn insights from the brain into timely
feedback that can effectively improve users' ability to learn
material.



<h4><em>3. Adaptive Experiences</em></h4>



The third killer application is a personalized, adaptive experience
directed by BCI. In gaming this is most immediate: a virtual hand with
fingers you can control (<strong>Furman et al 2016</strong>) and use as
easily as your own, an entire virtual body, an emotionally connected
avatar. Gaming content, audio content, video content, all might be able
to adaptively be tuned to individual users to improve their
experience.&nbsp;





For example, audio content can be played at variable speeds for podcasts
adapted to the user in the moment, and playlists can be customized to
achieve an effect like increasing user focus (<strong>Haruvi et al 2022</strong>). Arctop technology has already shown some success in this area, being
used to continuously measure a person's focus level while they listen to
audio content. This enables adaptive playlists that reliably increase
and sustain attention which is helpful for studying, working,
exercising, and many other essential human tasks that rely on focus.





In today's attention economy, information streams are increasingly rapid
and bite-sized, with distractions abounding. Technologies that help
maximize and maintain focus states can be invaluable. Using BCI
technology, applications could adjust experiences to theoretically
achieve any brain state aside from focus though, and that is perhaps the
most 'killer' part of this application. By anchoring applications
behaviors to objective measures of user cognition, a system of instant
feedback becomes available that has profound implications of what is
achievable for a user in concert with their BCI.&nbsp;



<h3>
<strong>Bioethics, Privacy &amp; Security</strong>
</h3>



Concerns about data ownership, data access rights, and data privacy
abound in BCI. Data may be processed on devices (edge computing),
locally, with user controlled data access rules and only sparse features
(SDK) being sent out encrypted to the Internet and Cloud or local
network. Or it could be processed like it's the cyber wild west with
unrestricted data access rules over real-time and historic data.&nbsp;





Markets, leading companies and government regulation will likely be the
greatest determiners of how these issues evolve. Across the world it
will be interesting to see if there is consensus, or if divergent
attitudes shape BCI technology in different directions. For now all we
can do is build to the best principles possible, and design for privacy
and the world we want to see.&nbsp;&nbsp;





BCI data is sensitive by nature since it contains information about a
person’s identity (<strong>Kopito et al 2021</strong>), health and their
real-time mental status so it needs to be treated accordingly. Through
the cognitive and affective state data BCIs process, much of what people
consider the most fundamental parts of themselves are exposed and
because of that, to say the least, the field is ripe with ethical,
security, and privacy issues that invite contributions from experts.





Open questions in the field include how brain data is measured. For
instance does the device clearly label that brain sensors are embedded?
People at minimum should know if their brain data is being processed it
seems. If they consent for it to be, the next layer is how is it used by
applications? At Arctop we take a user privacy-centered approach, but
this is not yet the prevailing model as increasingly cloud-based
approaches are used for business models based on data being shared or
accessed by affiliates. Some companies ask users to 'donate' their data,
some are less straightforward. The dramatic differences such technology
and data architecture decisions have on the end user's mental privacy
and personal rights writ large cannot be overemphasized.





BCI as a class of technology thus raises unique, complex issues. These
involve human agency questions as well, since actions that BCIs perform
must be treated as an extension of the person's own. Unless of course,
the BCI made a mistake in decoding what the user wanted. For example if
a prosthetic arm punches someone, who is to blame if the user denies
that they made the BCI punch intentionally? At what point does a crime
become a crime is another corner one may end up in doing work on the
ethics in this space – does law enforcement need brain data to prevent
drunk driving or a crime of passion? The reader can go ahead and fill in
the blanks here in terms of other sci-fi tropes and subplots.&nbsp;





There is also a significant moral and societal question of BCI "haves"
and "have-nots" and who is entitled to access the technology and for
what purpose, since access to BCI technology could profoundly
differentiate populations from one another and create positive and
negative feedback loops across cultures if one has access to technology
that accelerates learning and improves health that the other does not. (<strong>Bavelier et al 2019</strong>).



<h3>
Toward a General BCI: Interface, Interaction, Intelligence
</h3>



<Figure src="/deep-dives/pics/deep-dives/five_levels_part2_figure7.webp" alt="BCI Evolution" caption={"Interface, interaction, intelligence (I3 model) of BCI evolution. Inspired and adapted from <strong>Gao et al 2021</strong>."} />





An interesting model is proposed by <strong>Gao et al 2021</strong>:
that generalized BCI technology would evolve through three stages:
interface, interaction, and intelligence. The field of BCI has
transformed from “interfacing”, one-way brain-to-computer control, to
“interaction”, two-way co-adaptation for both human brain and computer.
This transition highlights the importance of the “Write” or “Encoding"
path, compared to the conventional “Read” or “Decoding” direction. The
“Write” path could use interactive auditory or visual feedback for
adaptive learning experience in cognitive augmentation or skill
learning, or it could use direct electrical stimulation as
neuromodulation therapies or neurorehabilitation, as described in <em>Level 4 Application</em>. Since learning is a master tool we believe that the killer
applications of read and write technology will be in the area of
teaching ourselves new things not just faster, but more memorably and
effectively.





The era of super-human AI “agents” and as they become the new
“computer”, the final stage of generalized BCI technology will be
“collaborative intelligence” – a seamless integration and collaboration
of human intelligence (HI) and artificial intelligence (AI) where
learning is not needed in many instances since the partnership is
coupled so tightly. HI is better at understanding, reasoning,
generalizing, empathizing, and goal-setting, while AI is better at
perceiving, memorizing, computing, interpreting, and achieving specific
tasks. AI can couple a user's goal-directed intents (e.g. attention,
preference) as reward functions in its reinforcement learning to align
goals. AI can integrate a user's states (e.g. emotions, stress) to
jointly make personalized, situational decisions in perfect harmony with
goals aligned.



<h3>
The Why: Building A General BCI
</h3>



A General BCI is one that works for any human immediately out of the box
with perfect decoding of all the elements and dimensions of a person's
Cognition. It is a system that has mapped the human experiential space
and has a unified informational model of how human's feel and understand
things - a symbiotic relationship where the BCI does everything it can
to maximize the user's quality of life. At scale, General BCI systems in
use by a population of people through the same approach should be able
to maximize the quality of life of the community as a whole.





By making quality of life a quantified, data driven metric to be
optimized by a General BCI the way can be charted towards improvement
for anyone. Outgrowing certain human limitations and suffering with
improved emotional regulation, reversing cognitive decline, maintaining
memory and healthy forgetting, downloading skills into your brain to
speak a new language. All are in the province of a General BCI, which by
necessity requires leaps forward in imagination and understanding of the
brain, human experience, and the environment in which both operate.





In the recent book "The End of Reality," writer Jonathan Taplin raises
the provocative point of the philosophy of the Greek Epicurus, which he
ascribes to being the opposite of what 'Big Tech' aims for today in that
for Epicurus the three elements of a happy life were unaddressable with
modern technologies:



<ol role="list">
<li>The company of good friends</li>
<li>The freedom and autonomy to enjoy meaningful work</li>
<li>An 'examined life'</li>
</ol>



We would argue to the contrary that these Epicurean values are actually
best addressed with a BCI – not other tech or by 'Big Tech' per se, we
agree, but specifically BCI because of how it works – yet another
example of how this technology is in a unique class. Perhaps instead of
new communications, skills learning and adaptive experiences, those
three elements above should be the anchors for BCI.&nbsp;





Company of good friends can be supported well by an empathic application
powered by BCI for match making and scheduling and reviewing the
goodness of the friend fit with objective data plus the patience and
organization skill to maintain and cultivate friendships over time.
Freedom and autonomy to work is about communication, control, about not
being stopped from pursuing purposeful work regardless of where one is
or who they are. And an 'examined life' is another word we feel for
quantified self- the tech trend for which BCI is a type of apotheosis,
since it allows the most high granularity examination of one's body and
brain. So it may be that with BCI applied right, even the greatest
Luddites among us may be won over.





To create the ultimate General BCI that helps humans everywhere live a
higher quality of life, at Arctop we are focusing on the software
platform for decoding and building it in a hardware-agnostic way to be
as universal in connective applications as possible.&nbsp;Quality of
life for humans everywhere is our north star and to enable as many
people to enjoy a high quality of life we believe the brain, and BCI
technology, are the master tools. We can't do it alone since so much is
required in the technological and societal stack to bring BCI out
widely. Together we are at a unique time in human history with BCI, AI,
and VR technologies knitted together by an ever more powerful compute
that is near miraculously bringing us all into the General BCI era. With
Arctop software and developer tools we aim to accelerate that arrival.





Sometimes it's worth taking a moment to marvel at how far humans have
come. Here at the end of 2023, the end of this post, we are grateful
you're here reading and we are reflective on the distance run. But
mostly we are looking ahead — excited to release a new product early in
the new year that takes a leap forward to that goal of quality of life
for all. If you want a sneak peak at our baby General BCI, reach out to
us and we'll do our best to connect!





<Figure src="/deep-dives/pics/deep-dives/five_levels_part2_figure8.webp" alt="BCI Evolution" caption={"Proposed by the authors. Conceptual portrayal of how BCI may influence evolution."} />





<References items={[{"html":"Gao, X., Wang, Y., Chen, X. and Gao, S., 2021. Interface, interaction, and intelligence in generalized brain–computer interfaces. <em>Trends in cognitive sciences</em>, <em>25</em>(8), pp.671-684."},{"html":"Brunner, C., Birbaumer, N., Blankertz, B., Guger, C., Kübler, A., Mattia, D., Millán, J.D.R., Miralles, F., Nijholt, A., Opisso, E. and Ramsey, N., 2015. BNCI Horizon 2020: towards a roadmap for the BCI community. <em>Brain-computer interfaces</em>, <em>2</em>(1), pp.1-10."},{"html":"Chang, C.Y., Hsu, S.H., Pion-Tonachini, L. and Jung, T.P., 2019. Evaluation of artifact subspace reconstruction for automatic artifact components removal in multi-channel EEG recordings. <em>IEEE Transactions on Biomedical Engineering</em>, <em>67</em>(4), pp.1114-1121."},{"html":"Hsu, S.H., Pion-Tonachini, L., Palmer, J., Miyakoshi, M., Makeig, S. and Jung, T.P., 2018. Modeling brain dynamic state changes with adaptive mixture independent component analysis. NeuroImage, 183, pp.47-61."},{"html":"Hsu, S.H., Lin, Y., Onton, J., Jung, T.P. and Makeig, S., 2022. Unsupervised learning of brain state dynamics during emotion imagination using high-density EEG. NeuroImage, 249, p.118873."},{"html":"Chiang, K.J., Wei, C.S., Nakanishi, M. and Jung, T.P., 2021. Boosting template-based SSVEP decoding by cross-domain transfer learning. Journal of Neural Engineering, 18(1), p.016002."},{"html":"Singh, F., Shu, I.W., Hsu, S.H., Link, P., Pineda, J.A. and Granholm, E., 2020. Modulation of frontal gamma oscillations improves working memory in schizophrenia. <em>NeuroImage: Clinical</em>, <em>27</em>, p.102339."},{"html":"Zeller, C.J., Züst, M.A., Wunderlin, M., Nissen, C. and Klöppel, S., 2023. The promise of portable remote auditory stimulation tools to enhance slow‐wave sleep and prevent cognitive decline. Journal of sleep research, p.e13818."},{"html":"Zander, T.O. and Kothe, C., 2011. Towards passive brain–computer interfaces: applying brain–computer interface technology to human–machine systems in general. <em>Journal of neural engineering</em>, <em>8</em>(2), p.025005."},{"html":"Kopito, R., Haruvi, A., Brande-Eilat, N., Kalev, S., Kay, E. and Furman, D. 2021. <a href=\"https://www.biorxiv.org/content/10.1101/2021.04.09.439244v2.abstract\" target=\"_blank\">Brain-based Authentication: Towards A Scalable, Commercial Grade Solution Using Noninvasive Brain Signals</a>. bioRxiv, 2021.04. 09.439244"},{"html":"Haruvi, A., Kopito, R., Brande-Eilat, N., Kalev, S., Kay, E. and Furman, D. 2022. <a href=\"https://www.frontiersin.org/journals/computational-neuroscience/articles/10.3389/fncom.2021.760561/full\">Measuring and modeling the effect of audio on human focus in everyday environments using brain-computer interface technology</a>. Frontiers in Computational Neuroscience 15, 760561"},{"html":"Furman, D., Reichart, R. and Pratt, H. 2016. <a href=\"https://ieeexplore.ieee.org/abstract/document/7457445\" target=\"_blank\">Finger flexion imagery: EEG classification through physiologically-inspired feature extraction and hierarchical voting</a> 4th International Winter Conference on Brain-Computer Interface (BCI), 1-4"},{"html":"Furman, D., Benisty, H., Abramovich, T., Ivry, A., Pratt, H. 2016. <a href=\"https://ieeexplore.ieee.org/abstract/document/7806083\" target=\"_blank\">Enhancement of BCI classifiers through domain adaptation</a>. IEEE International Conference on the Science of Electrical Engineering."},{"html":"Furman, D. <a href=\"https://www.wsj.com/articles/computers-will-soon-read-your-mind-eeg-als-parkinson-brainwaves-560e5951?reflink=desktopwebshare_permalink\" target=\"_blank\">Computers Will Soon Read Your Mind: Technology will help patients suffering from ALS or strokes.</a>2023. The Wall Street Journal."},{"html":"Furman, D., Kwalwasser, E., 2023. <a href=\"https://patents.google.com/patent/US11662816B2/en\" target=\"_blank\">Interactive electronic content delivery in coordination with rapid decoding of brain activity</a>."},{"html":"Furman, D., Kwalwasser, E., 2021. <a href=\"https://patents.google.com/patent/US20210390366A1/en\" target=\"_blank\">Empathic Computing System and Methods for Improved Human Interactions With Digital Content Experiences.</a>"},{"html":"Lotte, F., Larrue, F., M̈uhl, C. 2013. Flaws in current human training protocols for spontaneous Brain-Computer Interfaces: lessons learned from instructional design. Frontiers in Human Neurosciences, Frontiers, 7 (568)"},{"html":"Bavelier, D., Savulescu, J., Fried, L., Friedmann, T, Lathan, C., Schürle, S., Beard, J,. Rethinking Human Enhancement as Collective Welfarism. Nat Hum Behav. 2019 Mar; 3(3): 204–206. Published online 2019 Feb 11. doi: 10.1038/s41562-019-0545-2"},{"html":"Yerkes RM, Dodson JD (1908). \"The relation of strength of stimulus to rapidity of habit-formation\". Journal of Comparative Neurology and Psychology. 18 (5): 459–482. doi:10.1002/cne.920180503."},{"html":"Taplin, J. 2023. The End of Reality: How Four Billionaires are Selling a Fantasy Future of the Metaverse, Mars, and Crypto. Publisher: PublicAffairs. ISBN: 9781541703155"}]} />

---

# Computers Will Soon Read Your Mind
URL: https://arctop.com/deep-dives/computers-will-soon-read-your-mind
Published: 2023-12-14

<p class="dd-post__source-note"><em>Originally published in <a href="https://www.wsj.com/opinion/computers-will-soon-read-your-mind-eeg-als-parkinson-brainwaves-560e5951" target="_blank" rel="noopener noreferrer">The Wall Street Journal</a> on December 14, 2023.</em></p>

It's been almost a century since psychiatrist Hans Berger made the first electroencephalogram, providing a glimpse into the electric nature of the human brain. EEG readings have helped countless people struggling to recover from ailments ranging from epilepsy and sleep disorders to head injuries and brain tumors. Technology has come a long way since then, and artificial intelligence may soon give us a new brain technology revolution, with advances in the treatment of ALS, strokes and other conditions.

As a teenager in a mentorship program, I decided to study the brain after watching a neurosurgeon implant an electrode deep into the brain of a patient with Parkinson's whose tremors were making it impossible for her to hold a pen or drink from a cup. The surgeon implanted the electrode — designed to deliver the right amount of electricity to the exact part of the brain responsible for the tremors — and awoke the patient, her skull still open, to adjust the implant's settings. A few turns of a dial and the shaking stopped. Her tremors were cured.

While the discovery of EEG signals was revolutionary, they can be noisy and difficult to interpret, requiring expensive equipment and controlled environments. With recent advances in sensor materials, we are approaching the point at which brain signals can be read throughout the day with comfortable and discreet wearable devices, as a Fitbit or Apple Watch measures our heart rates. Advances in computing and AI mean we could interpret these brain signals in real time.

The possibilities include thought-to-speech and thought-to-movement assistive technology for ALS or paralysis patients and accelerated, customized recovery protocols for those suffering from strokes, post-traumatic stress disorder and brain trauma. Brain-computer interfaces could also help personalize teaching and training protocols to fit a learner's cognition and memory processes, eliminate the need for usernames and passwords with a seamless "brain ID," and enable you or a mental-health professional to monitor your emotional state throughout the day.

I was part of the team that tried to develop a brain-computer interface for the astrophysicist Stephen Hawking, who suffered from ALS. Hawking's Intel-designed eye-tracking and cheek-click method relied on a level of muscular control that couldn't be taken for granted given his condition. He participated in the project, as he put it, "to assist in research, encourage investment in this area, and, most importantly, to offer some future hope to people diagnosed with ALS and other neurodegenerative conditions." He died in 2018.

Today implant-based systems are increasingly powerful and noninvasive, and wearables are improving quickly too. Many of us in the field believe we are nearing an inflection point when countless people will see the fruits of decades of research. The stakes are high. Although every new technology carries promises and risks, few are tied so intimately with who we are.

<p><em>Mr. Furman is a founder and CEO of Arctop, which makes brain-decoding software.</em></p>

<p>Source: <a href="https://www.wsj.com/opinion/computers-will-soon-read-your-mind-eeg-als-parkinson-brainwaves-560e5951" target="_blank" rel="noopener noreferrer">The Wall Street Journal — "Computers Will Soon Read Your Mind"</a></p>

---

# Five Levels of Explanation (Part I): How Brain-Computer Interfaces Work
URL: https://arctop.com/deep-dives/how-bci-works-part-1
Published: 2023-12-14

What do you think of when you read "Brain-Computer Interfaces" in the title? Elon Musk's Neuralink? Or perhaps you imagined interacting in a virtual world like Neo in "The Matrix" or controlling a Na'vi body in "Avatar"? Media, science-fiction novels, and movies have popularized the concept of "brain interface" technology (for inspiration, consider this <a target="_blank" href="https://en.wikipedia.org/wiki/Category:Fiction_about_brain%E2%80%93computer_interface">list of BCI in fiction</a>), but they have also often led to misunderstandings. Our company prefers the term "Cognition Technology." We'll leave that for a later post.



  

The field of brain-computer interfaces (BCI) – though deeply rooted in science fiction stories and fantasy – is now a solidly established technical field which blends hard science and applied engineering, and is rapidly becoming a mainstream consumer technology after decades of research. BCI is jumping out of the lab and into ubiquity, just as this year the technical field celebrates the 50th anniversary of the coining of the term "BCI" by UCLA researcher Jacques Vidal in 1973 (Vidal 1973).



  <section class="bci-summary" aria-labelledby="what-is-bci">
    <h2 id="what-is-bci">What is BCI?</h2>
    

A brain-computer interface is a system that measures brain activity, decodes patterns with software, and translates those signals into useful outputs such as metrics, commands, or contextual feedback. Modern BCIs pair non-invasive sensors—like EEG headphones or earbuds—with AI models that run on edge or cloud infrastructure. The goal is to turn cognitive state, intent, or biometric signatures into actionable data in real time while keeping the experience comfortable and safe for everyday use.


    

Every BCI pipeline follows the same loop: <strong>sense</strong> neural signals, <strong>process</strong> them with algorithms, and <strong>deliver</strong> feedback to the user or attached device. Improvements in wearable hardware, signal processing, and machine learning are what make Arctop’s cognition metrics and developer tools possible without implants.


  </section>

  

In this post, we aim to help the curious understand what BCI is, explore its capabilities, debunk overhyped claims, and understand its potential impact on our society. Consider this as an invitation to join us in contemplating the future of this technology, its boundaries, emerging opportunities, and how you can be a part of it.


  
  

Finding answers to these questions is not an easy task. Even among the BCI community, there is disagreement on priorities, opportunities, and capabilities. It's also crucial to differentiate between theoretical possibilities and practical limitations. Just because something can work in theory does not mean it will work in practice.



  

It's vital for us, amidst this technological surge, to establish common ground for future conversation that extends beyond research labs and pop culture, to addressing humanity's broader mission to better itself and possibly co-adapt with artificial intelligence.



  

In this spirit, akin to <a target="_blank" href="https://www.youtube.com/results?search_query=wired+five+levels">WIRED's</a> style, we'll explain brain-computer interfaces in five levels of difficulty. We hope you find reading it both fun and enlightening! Let's dive into levels 1 through 3.



  <h2 id="level-1">Level 1: Child</h2>
  
  

Imagine telling a computer what to do just by thinking! That's what BCI is about.



  

You don't have to use your hands to type, press any buttons, or even talk. Just put on a cool gadget on your head, like a headband or headphones or special earbuds and these BCI gadgets can "hear" what your brain is thinking and follow your commands.



  

What you can do with this "magic" is only limited by your imagination!



  

<Figure src="/deep-dives/pics/deep-dives/five_levels_part1_figure1.webp" alt="Child Illustration of Brain-Computer Interface System" caption={"Figure 1: AI-generated image by the authors"} />



  <h2 id="level-2">Level 2: Teen</h2>
  
  

BCIs offer a revolutionary way to interact with computers by translating your brain's impulses into commands. You think about an action, and it happens. All you need to do is to wear a special sensor. BCI sensors can even pick up on your emotions, allowing different apps to personalize your experience in amazing ways. Want to search the Internet, send a message, or play your favorite song? Simply think about it.



  

While this might sound like something out of a sci-fi movie, the past decade has seen incredible strides forward in this field. A blend of advanced scientific knowledge, engineering capabilities, and software development has turned what once seemed like a distant dream into reality.


  

Let's try to understand BCI with a fun analogy. Imagine a football stadium filled with people. Now, picture your brain as this stadium. A bit weird, but you got it, right? If not, the AI-generated image below might help.



  

<Figure src="/deep-dives/pics/deep-dives/five_levels_part1_figure2.webp" alt="Brain as a Football Stadium" caption={"Figure 2: AI-generated image by the authors"} />



  

In this scenario, each person in the stands is like a single brain cell, called a neuron. As they watch the game, they cheer, shout, chant, holler and sing based on what's happening on the field. BCI sensors capture your brain's activity similar to how microphones placed outside the stadium would pick up the crowd's reactions.



  

Of course, from these "outside" measurements, there are limits. You can't hear what everyone is saying, but you can definitely get the general vibe of the game and what the score is. A loud roar might mean a goal; sudden gasps could indicate a surprising play. You might also pick up from the recordings a "wave" ripple through the crowd or the building excitement during a tense play that ends with one team's fans singing their anthem.



  

Similarly, while a BCI can't decipher every neuron's activity or every single thought in your brain, it can detect changes and patterns in your brain at a level that reflects your thoughts and moods - like whether you're focusing, feeling sleepy, or thinking about moving your hand. Think of it like this: the microphones outside can capture big moments in the game just like BCI sensors can pick up the broader patterns in your brain.



  

You might be wondering, 'if BCI can do so many things, why aren't my friends using it yet?' The best answer we have is that, like the Apple Vision Pro, it's right around the corner. Thanks to advancements in scientific knowledge, better sensors, and more powerful computers with AI, new BCI products are starting to emerge. For now, and for a variety of reasons, BCIs are currently in use by relatively few people: mostly developers, researchers or individuals with disabilities. For example, they help people who've lost the ability to speak or move due to a stroke or certain diseases.



  <h2 id="level-3">Level 3: College Student</h2>

  

<em>
      "The most profound technologies are those that disappear. They weave themselves into the fabric of everyday life until they are indistinguishable from it."
    </em>
    <br />
    Mark Weiser, 1991.


    
  

BCI, originally developed for medical use, is rapidly evolving into a significant technology for everyday life. A typical BCI system includes three components: the hardware that measures brain activity, the software that interprets this data, and the mechanism that enables interaction with external device(s) and provides user feedback (proposed by Wolpaw et al 2002, one of the foundational papers that pioneered BCI). You can liken these components to the processes of perception, comprehension, and action – what a computer or AI agent needs to interact with you.



  

<Figure src="/deep-dives/pics/deep-dives/five_levels_part1_figure3.webp" alt="BCI System Components" caption={"Figure 3: Classical diagram of the three components of a BCI system. Image by the authors."} />



  

There are various brain-sensing modalities and corresponding hardwares to measure brain activity. These include electroencephalography (EEG), electrocorticography (ECoG), and local field potential (LFP) for electrical activity (see the image below); magnetoencephalography (MEG) for magnetic activity; and functional magnetic resonance imaging (fMRI) and functional near-infrared spectroscopy (fNIRS) for blood oxygenation (see Table 1 in Saha et el 2021 for a summary and comparison across these modalities). Among these, EEG – which records the brain's electrical activity by placing sensors on your scalp – is most prevalent in BCIs due to its safety, comfort, fast tracking of the dynamic brain, and affordability.



  

<Figure src="/deep-dives/pics/deep-dives/five_levels_part1_feature.webp" alt="Modalities for measuring brain electrical activity" caption={"Figure 4: Modalities for measuring brain electrical activity: from non-invasive techniques like EEG to more invasive methods like ECoG and LFP. [Adapted from Thakor, 2013]"} />



  

Revisiting our football stadium analogy, consider BCIs from EEG sensors outside the head to be similar to microphones placed outside the stadium. One way to glean more information about what is happening inside is you can increase the number of microphones (EEG sensors) around various parts of the stadium (the brain). This strategy, "increasing spatial sampling," enhances the detail and robustness of brain activity data, and correspondingly enhances the decoding capabilities of the BCI system. With more comprehensive data, BCI developers can employ sophisticated brain signal processing and machine learning techniques to decode and understand more intricate patterns of brain activity.



  

Now let's delve into what BCIs can actually decipher from brain activity measured via EEG. This technology, akin to an EKG for the brain, has been around for nearly 100 years. EEG continuously captures the brain's activity in tiny voltage fluctuations on the skin, reflecting various experiences – from sensory reactions and movement intention to cognitive processes like attention, memory, and decision making.



  

For instance, when you look at a flashing light, your brain's visual area (occipital lobe) produces oscillatory activities (steady-state visual evoked potentials, or SSVEPs) that reflect the light's flashing frequency. This phenomenon is often described as the brain "entraining" to the stimulus. Similarly, rhythmic sounds produce corresponding activities (auditory evoked potentials, AEPs) in the auditory area (temporal lobe). These responses not only mirror the beats but also the melodies and emotional content in the music. Even imagining movements, like thinking about moving your left arm, induce identifiable changes (event-related desynchronization, ERD) in the motor planning area (parietal lobe). These diverse signals, all detectable by EEG, have been instrumental in advancing BCI applications over the last two decades.



  

<Figure src="/deep-dives/pics/deep-dives/five_levels_part1_figure4.webp" alt="EEG signals from different brain regions" caption={"Figure 5: Differentiating left (red) and right (green) arm imagery movement in the brain: EEG readings show a negative deflection (event-related desynchronization, ERD) in the brain's opposite side. The EEG sensors C3 (left panel) and C4 (right panel) are positioned near the motor planning area on the brain's left and right sides, respectively. [Adapted from Blankertz et al., 2008]"} />



  

EEG can also reveal different brain states. For instance, closing your eyes induces an instant and distinct power increase in oscillations at around 10 cycles per second in your visual cortex. Sleep stages, from light to deep sleep, are marked by increasingly synchronized, broad, and slow-oscillating brain activities (known as slow waves, oscillating at just a few cycles per second). Interestingly, the EEG patterns during dreaming resemble those of wakefulness. Recent studies have further expanded EEG's scope, demonstrating its ability to capture various mental states, from changes in attention and emotional states to varying levels of drowsiness.



  

The brain activities captured by EEG are subtle, often only a few microvolts in amplitude. They're easily overshadowed by non-brain activities like muscle or eye movement, which can be ten or hundred times stronger. This complexity demands that BCI developers have a deep understanding of signal processing and machine learning to accurately interpret the faint EEG data, especially in real-time, real-world scenarios.



  

<Figure src="/deep-dives/pics/deep-dives/five_levels_part1_figure5.webp" alt="EEG signals and patterns" caption={"Figure 6: Different brain states and their corresponding EEG patterns. [Adapted from Budzynski, et al., 2009]"} />



  

Over the past two decades, we have started to see more developments and applications for mainstream uses, from enhancing computer games and virtual-reality experiences, providing feedback to promote meditation training, tracking health metrics like sleep, and integrating into learning with content adapted to student's mental states. In our next post, we'll go into details at a graduate level into how BCIs work and are increasingly advancing into the fabric of life.



  

Levels 4 and 5 coming soon…


<section class="bci-faq" aria-labelledby="bci-faq-heading">
  <h2 id="bci-faq-heading">Brain-Computer Interface FAQ</h2>
  <dl>
    <dt>What is a brain-computer interface?</dt>
    <dd>A brain-computer interface (BCI) is a system that measures brain activity, decodes it with software, and turns those signals into outputs such as metrics, commands, or feedback for a connected device.</dd>
    <dt>Do BCIs require surgery or implants?</dt>
    <dd>Most consumer and enterprise BCIs today are non-invasive. They use wearable sensors like EEG headsets or earbuds to capture brain activity through the scalp, so no surgery is required.</dd>
    <dt>What can BCIs do today?</dt>
    <dd>Modern BCIs can monitor cognition metrics such as focus or engagement, enable adaptive software experiences, support accessibility use cases, and provide biometric authentication based on unique brain signal signatures.</dd>
  </dl>
</section>


<References items={[{"html":"J. J. Vidal, \"Toward direct brain-computer communication.,\" Annu. Rev. Biophys. Bioeng., vol. 2, pp. 157–80, Jan. 1973. https://www.annualreviews.org/doi/10.1146/annurev.bb.02.060173.001105"},{"html":"Jonathan R Wolpaw, Niels Birbaumer, Dennis J McFarland, Gert Pfurtscheller, Theresa M Vaughan, \"Brain–computer interfaces for communication and control\", Clinical Neurophysiology, Volume 113, Issue 6, 2002, Pages 767-791, ISSN 1388-2457, https://doi.org/10.1016/S1388-2457(02)00057-3."},{"html":"Saha Simanto, Mamun Khondaker A., Ahmed Khawza, Mostafa Raqibul, Naik Ganesh R., Darvishi Sam, Khandoker Ahsan H., Baumert Mathias, \"Progress in Brain Computer Interface: Challenges and Opportunities\", Frontiers in Systems Neuroscience, Vol 15, 2021, https://www.frontiersin.org/articles/10.3389/fnsys.2021.578875"},{"html":"Nitish V. Thakor, \"Translating the Brain-Machine Interface.\" 2013. Science Translational Medicine, Vol 5 No 210, Available at https://www.science.org/doi/10.1126/scitranslmed.3007303"},{"html":"B. Blankertz, R. Tomioka, S. Lemm, M. Kawanabe, and K. Muller, \"Optimizing Spatial filters for Robust EEG Single-Trial Analysis,\" IEEE Signal Process. Mag., vol. 25, no. 1, pp. 41–56, 2008."},{"html":"T. Budzynski, H. Budzynski, J. Evans, and A. Abarbanel, \"Introduction to quantitative EEG and neurofeedback: Advanced theory and applications.\" 2009."}]} />

---

# Brain-based Authentication: Towards A Scalable, Commercial Grade Solution Using Noninvasive Signals
URL: https://arctop.com/deep-dives/brain-based-authentication
Published: 2021-04-09

<div class="rich-text-block-2 w-richtext">


This paper is available in PDF format <a href="https://www.biorxiv.org/content/10.1101/2021.04.09.439244v2.full.pdf">here</a>.



<h2>Abstract</h2>


Here we report on a field test where we asked if it is feasible to
deliver a scalable, commercial-grade solution for brain-based
authentication given currently available head wearables. In this
study, forty-nine (49) participants completed multiple sessions in
their natural home environment over a single week. Participants used
an off-the-shelf brain signal measuring headband to record their own
brain activity while completing various tasks. Recording sessions
were self-operated by the participants and unsupervised by any
expert or technician to simulate real world use cases, while also
contrasting common research approaches to this topic that rely on
data from controlled laboratory conditions. Although brain signals
have a non-stationary, complex nature, when participants watched
rapidly presented images, our authentication system was able to
successfully construct a unique and robust "brain ID" for each
participant. Based on this brain ID, we developed a simplified
brain-based authentication method that captures distinguishable
information with reliable, commercial-grade performance from
participants at their own homes. We conclude that noninvasively
measured brain signals are ideal for use in biometric authentication
systems, especially in environments where head wearables such as
headphones or AR/VR devices are used as these devices offer a
natural form factor for capturing participant brain ID continuously.


<h2>Introduction</h2>


When a user requests access to a system or a device, an
authentication process must confirm whether the identity claim of
the user is genuine or whether they are an imposter. A simple
example of an authentication method is an alphanumeric password like
'abc123,' while a more complex example is a digital fingerprint
captured by a smartphone sensor. Effective authentication is
critical to security for both consumers and enterprises. Because of
the high frequency of use of authentication systems, methods need to
be both convenient and secure. That balance - between convenience
and security - is a defining performance characteristic of
authentication systems. The strongest authentication systems are
very secure and, often, very cumbersome to implement and maintain.
In contrast, weak authentication methods are very convenient, but
have been responsible for countless data breaches because of their
equivalent ease of being hacked. The "password chaos" of modern life
seems to have reached a boiling point and it is clear that future
computing systems need improved methods that both deliver increased
security along with an increased convenience that ensures adherence,
at the human level of behavior, to security protocols.




Biometric authentication is any method that uses natural occurring
information to verify a user's identity. Many biometric
authentication systems have already been developed based on
fingerprints, faces, palm veins, irises, voices, gaits, and other
metrics. There are significant advantages to using biometrics for
authentication, since user experience is convenient and fast, it is
non-transferable, and usually the system can reach very high
performances (high false rejection rate, and low false acceptance
rate) typically without requiring much attention, if any, from the
user.




There are disadvantages as well however, since biometrics can be
faked or stolen and when that happens, the victim cannot simply
replace them to avoid impersonation. Certain biometrics, for example
fingerprints or faces, can be easily captured today by cameras
remotely without any knowledge from the individual that is being
surveilled, and once a face or finger ID is compromised, the remedy
is extremely difficult.




Brain-based authentication is the process of verifying an
individual's identity by using their brain signal, and as an
approach it offers several distinct advantages over other biometric
authentication methods. Since at least the late 1980s<strong><sup>1</sup></strong> neuroscientists have observed that noninvasively measured human brain
signals carry personally identifying information<strong><sup>2</sup></strong><sup>,</sup><strong><sup>3</sup></strong> that differentiates between
family members and across a broad population.<strong><sup>4</sup></strong><sup>,</sup><strong><sup>5</sup></strong> Brain signals, unlike many
other biomarkers are concealed: an invisible signal that is never exposed
in daily life. Second, brain signals are dynamic, non-stationary and
extremely complex. They are the result of a unique series of brain waves
superpositioning in a given brain at any moment, and these waves reflect
both personal brain function and anatomy. Taken together, this makes
brain signals an ideal candidate for use as a biometric<strong><sup>6</sup></strong><sup>,</sup><strong><sup>7</sup></strong><sup>,</sup><strong><sup>8</sup></strong> method. Indeed, many groups have attempted to build biometric authentication
systems based on brain signals.<strong><sup>9</sup></strong><sup>,</sup><strong><sup>10</sup></strong><sup>,</sup><strong><sup>11</sup></strong><sup>,</sup><strong><sup>12</sup></strong><sup>,</sup><strong><sup>13</sup></strong><sup>,</sup><strong><sup>14</sup></strong><sup>,</sup><strong><sup>15</sup></strong><sup>,</sup><strong><sup>16</sup></strong><sup>,</sup><strong><sup>17</sup></strong><sup>,</sup><strong><sup>18</sup></strong><sup>,</sup><strong><sup>19</sup></strong> Generally, the process involves a machine-learning classifier to identify
if a given brain signal belongs to a genuine identity or to an imposter
one (<strong>Figure 1</strong>).




<Figure src="/deep-dives/pics/deep-dives/brain_based_auth_feature.webp" alt="Schematic illustration of the brain-based authentication process." caption={"<strong>Figure 1: Schematic illustration of the brain-based authentication process.</strong> Brain signal is recorded while participants watch images (\"Prompt\"). The brain response (e.g. \"John's brain signal\") is fed into a trained classifier of that participant. The classifier decides if the brain pattern matches the participant (authenticates \"John\") or not (rejects)."} />




The overall usability of brain-based authentication systems has been
increasing since 2010,<strong><sup>20</sup></strong> however most are
still far from proving field-viability and bringing new value and utility
to existing authentication providers.<strong><sup>21</sup></strong><sup>,</sup><strong><sup>22</sup></strong> Brain data is commonly collected
in a laboratory under controlled conditions in other studies, where a
trained technician is an essential part of the brain measurement procedure.
Often authentication performance tests are also done with a small number
of people, with all measurements taking place in a single session despite
the common industry knowledge that single session data maps poorly to
daily consumer electronics usage, where individuals put devices on and
off regularly.<strong><sup>23</sup></strong><sup>,</sup><strong><sup>24</sup></strong><sup>,</sup><strong><sup>25</sup></strong><sup>,</sup><strong><sup>26</sup></strong>




While brain biometric identity appears to be one of the most
natural, powerful methods for head wearables, its robustness has not
been sufficiently vetted in real world conditions that parallel the
end use cases such as:

<ul role="list">
<li>
Professionals who work remotely and wear headsets as part of
their daily job, which requires them to be authenticated across
different applications throughout the day.
</li><li>
Gamers who want a seamless, hands-free and voice-free method of
profile loading and authorizing in-game purchases.
</li><li>
e-Commerce consumers whose check-out experience is currently
interrupted by passwords.
</li><li>
Surgeons, heavy machinery operators, medical professionals and
others working in high-strain, sanitary environments that
require especially high reliability and convenience.
</li><li>
Air gapped environments where there are strict demands on
performance, confidentiality and all biometric processing and
decisions need to be performed on-edge devices.
</li>
</ul>


To model these use-cases, we set out to perform a generalizable
field test of brain-based authentication using brain signals
measured noninvasively from people in their natural home
environment. In other words, their real world context. This
feasibility test "in the wild" advances the applied science of brain
biometric analysis towards scalable implementations as all
participants were completely new (naïve users) to the system and
enrolled themselves from home. They used a comfortable head wearable
for hours during test sessions, and this device had minimal data and
battery requirements. They performed repeated authentication
attempts across several different days, and individual brain ID's
were shown to be robust against changes in brain state and ambient
noise inherent to brain data.


<h2>Methods</h2>
<h3>
Participants
</h3>


Sixty-two (62) participants were recruited to complete four (4)
sessions over a single (1) week at their own home. Adult
participants were recruited from an opt-in screening panel and came
from all five (5) major regions of the continental United States
(Northeast, Southwest, West, Southeast, and Midwest). Only
participants who reported normal vision, or vision that was
corrected to normal with contact lenses were included. We excluded
volunteers who reported using medication that might influence the
experiment or other neurological or psychiatric conditions that
could influence the results. Written informed consent was obtained
from all participants before screening and the main experimental
sessions. Thirteen (13) participants were ultimately excluded for
problematic survey response patterns within the study and/or invalid
brain data, leaving 49 participants (mean age= 36, SD=8.25, 16
females) enrolled and eligible to be included in the analysis.


<h3>Sessions</h3>


Individuals participated in the study by recording sessions from
their own homes at their own pace, over one week as detailed in
Haruvi et al 2021.<strong><sup>27</sup></strong> Each participant received
an Arctop technology kit that included headphones (Sony), a brain signal
measuring headband (InteraXon) and a tablet computer (Samsung) with a
designated app (Arctop). Each participant recorded four sessions, one
hour long each where towards the end of each session, six (6) authentication
events were presented. Each authentication event started with a message
declaring the upcoming event and instructing the participant to stay
steady. Then, a fixation period which enabled the participant to get
prepared (2 seconds) before seeing the rapid serial visual presentation
(RSVP) of selected images at 10Hz for 10 seconds (<strong>Figure 2</strong>).




<Figure src="/deep-dives/pics/deep-dives/brain_based_auth_figure2.webp" alt="Time course of brain-based authentication using rapid serial visual presentation prompts." caption={"<strong>Figure 2: Time course of brain-based authentication using rapid serial visual presentation prompts.</strong> At each authentication event a sequence of images rapidly (10Hz) changes for 10 seconds while the brain response is recorded. In each session, six RSVP events were presented to each participant."} />


<h3>Data Acquisition</h3>


Participants engaged in a variety of tasks during each session while
their electrical brain activity was recorded using InteraXon's
Muse-S device, a portable, noninvasive electroencephalography (EEG)
device weighing 41 grams (<strong>Figure 3</strong>, left panel).
The device includes four dry fabric EEG sensors (sampling rate: 256
Hz), photoplethysmography (PPG) sensors (for heart rate) and motion
sensors (gyroscope and accelerometer). The EEG sensors are located
on the scalp, two frontal channels (AF7 and AF8) and two temporals
which rest behind the ears (TP9 and TP10), with a reference channel
at Fpz. The headbands were put on by the participants themselves,
with the assistance of a Quality Assurance (QA) screen that started
before each session. The QA showed the participants, in real-time,
the channels' quality, easily directing them to adjust the headband
properly for optimal signal quality.




<Figure src="/deep-dives/pics/deep-dives/brain_based_auth_figure3.webp" alt="Schematic illustration of the processing pipeline." caption={"<strong>Figure 3: Schematic illustration of the processing pipeline. </strong>Data acquisition included at home brain data recordings of 4 sessions, each on a different day. Each session included 6 RSVP videos (<strong>Figure 2</strong>). Brain data processing included filtering the signal, feature extraction and training a machine learning authentication classifier per participant. The classifier decides if the input belongs to the participant (Genuine) or not (Imposter)."} />


<h3>Preprocessing and feature extraction</h3>


Data analysis was performed only for periods within the RSVP events.
A band-pass filter (0.5-46Hz) was applied on each channel. The
filtered signal of RSVP event was segmented into 31 epochs of 2
seconds in length, using a sliding window with a stride of 250ms 
(<strong>Figure 3</strong>, middle panels). Comprehensive feature extraction and engineering
was not the goal of this current study. Here we aimed for effective
information capture without deeper optimization to first test the
core principles. Accordingly, for each epoch and for each EEG
channel, the following features were calculated: The average power
for each of the traditional frequency bands (Alpha, Beta, Gamma,
Delta, Theta), power spectrum interactions (engagement index, Alpha
over Delta, Beta over Theta, Theta/Alpha), time domain features such
as averages, standard-deviations, kurtosis, entropy and number of
zero-crossing points, and pairwise correlations between channels for
the various frequency bands. All together, for each epoch, a total
of 140 features were extracted.


<h3>Models training and testing</h3>


For each participant we had a total of 24 RSVP events (Supp. Video
1), which we collected over the 4 sessions. For each participant,
three sessions (18 events) were chosen randomly to be used for
training (<strong>Figure 3</strong>, right panels). The fourth
session was used for testing (6 events). Authentication prediction
of an event acts in two steps, the first at the epoch level, where
each epoch is determined to belong to a genuine or imposter. Second
is the final decision regarding the whole event identity (genuine or
imposter).




For each participant an authentication model was trained first at
the epoch level. Model classification was done with XGBoost
classifier (binary classification). The labeling of the data was
changed in accordance to the identity under training. Epochs
(feature space, 140 features per epoch) of genuine identity were
labeled as one (558 epochs), while epochs from the rest of the
participants were of imposter identity, and labeled as zero (26784
epochs). Thirty percent of training data (random and balance split)
was dedicated for validation and to determine epoch thresholding.




Standardization procedure over the training epochs was applied.
Later, standardization means and stds (standard deviation values) of
the training features were used to normalize the validation and
testing data. Epoch's threshold for classification was optimized to
minimize false acceptance rate (FAR), while maximizing true
rejection rate (see <strong>Supp. Figure 3</strong>). Identity
predictions of validation data epochs, after thresholding, exhibited
high accuracy for all participants (average accuracy=0.9865,
STD=0.00929).




A final decision about participant identity was given at the event
level. Event threshold, as before, was determined by an optimization
algorithm, but here it was done over the validation data. For the
validation data, after thresholding, the average event
authentication accuracy over all participants was 0.9965, with
STD=0.00041. Since per each participant validation data included 264
events, it suggests that on the average, after thresholding all
events were identified correctly except one.




Test data included 294 events and 9114 epochs. Training model
predicted the identity probability of each epoch. Probabilities
above the epoch threshold were determined to be of a genuine
identity, while those below the threshold are of imposter identity (<strong>Supp. Figure 4A-C</strong>). Test event was declared to be of genuine identity only if 40% of
its epochs were above the epoch threshold (<strong>Supp. Figure 4A1-C1</strong>).


<h2>Results</h2>


In our authentication system we derive from non-invasively recorded
brain signals a "brain ID" abstraction that proved to be
representative of each participant, and differentiating from one
another. The brain response during a RSVP event is used as a brain
biometric ID for identity verification. In order to demonstrate
characteristics of this brain ID, we will follow the example
presented in <strong>Figure 4</strong>. In our system, the
authentication period depends on the event length, here it is
approximately 10 seconds (one RSVP event). The event is composed of
4 channels (256Hz), segmented into 31 epochs, 2 seconds long, with a
stride of 250ms (<strong>Figure 4A-B</strong>). The non-stationary
nature of the brain signal, and the fact that it is a superposition
of hundreds of simultaneous processes in the brain, makes the signal
unique in time, unrepeatable, and unpredictable. Even when a user's
brain is stimulated by identical stimuli, no two epochs are alike (<strong>Figure 4B</strong>), nor are two events alike. In <strong>Figure 4C</strong> and in <strong>Figure 4D</strong> the epochs of two events (the brain signal space) of the same participant
are presented respectively. For each event, the epochs are aligned vertically,
ordered in time, creating a visualization of the brain ID. This representation
enables us to see easily that none of the epochs are identical, nor are
the full events.




<Figure src="/deep-dives/pics/deep-dives/brain_based_auth_figure4.webp" alt="Event epoching." caption={"<strong>Figure 4: Event epoching. </strong><br />A. The RSVP authentication event is a normalized filtered brain signal response composed of four channels of EEG data, 10 seconds long. Each event signal is segmented into 31 epochs (where each epoch carries 4 channels), 2 seconds in length, and with a sliding window of 0.25 sec stride. B. Channel-1 of epochs #5, and #28 (top, bottom) are shown for demonstration. Note, that the shaded areas colored in green and red in panel A correspond for these epochs respectively. C. The epochs of the event signal in A, can be rearranged into an image (here again just channel-1 is shown). Where each row is an epoch, and the epochs are time ordered vertically. In C and D, events which were taken from the same participant (#39), but from different sessions are shown. E, F the corresponding features of the epochs presented in C, D are presented. Note that the calculation of epoch features involves all epoch channels. While the non-stationary nature of the EEG data dictates that the events (as shown in C, D) do not resemble each other, the features images (E, F) demonstrate high similarity."} />




In contrast, the brain ID data becomes highly correlated when the
same events shown previously in <strong>Figure 4C-D</strong> are now
presented at a higher level of analysis (the features space) (<strong>Figure 4E-F</strong>). High correlation is visible among epochs of the same event,
creating a clear brain ID pattern. As one can note, a similar
pattern is carried by brain data captured at different occasions,
and we can conclude that usage of an event instead of a single epoch
for deriving the brain ID increases the pattern robustness and
increases both the sensitivity (true acceptance rate) and
specificity (true rejection rate) of the system.




In <strong>Figure 5A-D</strong>, four brain IDs examples of
different participants are presented. It is apparent that each brain
ID carries a unique pattern that is distinguishable from the others.
We would like to generalize the idea of using RSVP events brain IDs
as a verification method in our authentication system over all the
participants. If the event brain ID is used as an identity
verification two criteria must be fulfilled:

<ol role="list">
<li>
The similarity between different events brain IDs of the same
person is kept high: even and especially, for events which were
recorded at different occasions.
</li><li>
The brain ID of each participant is unique, and distinguishable.
</li>
</ol>


<Figure src="/deep-dives/pics/deep-dives/brain_based_auth_figure5.webp" alt="Similarity among intra and across inter participants events." caption={"<strong>Figure 5: Similarity among intra and across inter participants events.</strong> Panels A, B, C, and D show the features of a single event for different participants (#23, #35, #,39, and #34 respectively). The pattern of an event appears more robust, as the features values are repeatedly conserved across many epochs. On the other hand, it looks like for each participant the pattern is specific. The similarity (or dissimilarity) between events can be measured by a correlation coefficient. In E we present the event correlation matrix, where element Eij, is the average pairwise correlation across all training events of participant i and participant j. Note that the intra-correlation coefficients (diagonal) are usually higher than inter-correlation (off-diagonal), suggesting that for the same participant the pattern of different events is conserved, and patterns of different participants are different. This understanding leads us to the idea of an authentication system by events. Also note that the order of the participants in E, is in accordance with the hierarchy cluster tree shown in <strong>Supp. Figure 1</strong>. The matrix here is normalized."} />




The similarity between two events (at the feature space) can be
measured by the Pearson correlation coefficient between the means of
the events. Thus the similarity between two participants is the mean
of all pairwise events correlations of these participants. In <strong>Figure 5E</strong>, the normalized correlation matrix across all participants is
presented. Values are represented by colors (colorbar 0.2-1), higher
values suggest higher similarity. The order of participants along
the axes was determined using a hierarchical clustering algorithm
(see <strong>Supp. Figure 1</strong>). The averaged similarity
between events belonging to the same participant (intra correlation)
are along the diagonal elements of the correlation matrix, while the
averaged similarity between events of two different participants
(inter correlation), are the off diagonal matrix. In general, we
have found that for all participants, the similarity of intra
correlation is higher than the inter correlation (<strong>Figure 5E</strong>, <strong>Supp. Figure 2</strong>).




Looking more deeply, histograms in <strong>Supp. Figure 2</strong>,
shows that most of the inter- and intra-participant correlation are
indeed separated: for more than half of the participants the
intra-correlation is higher than 0.7, where most of the
inter-correlations are lower than 0.35. The inset in <strong>Supp. Figure 2</strong> also suggests a linear relation between the mean inter-correlation
of a participant and its intra correlation. Namely, participants having
relatively low intra-correlation (~0.5), their inter-correlations will
be low as well (~0.25). These results reflect that criteria 1 and 2 (above)
are fulfilled, and the brain IDs can be used for identity classification.




As described in the Method section, for each participant an
authentication model and relevant thresholds were found. These
models were tested on the test data which in total included 249
genuine events, and 14112 imposter events. The general performance
of our authentication system is summarized in <strong>Figure 6</strong>. The averaged false acceptance rate (FAR) is 9% and the false
rejection rate (FRR) is 13%, making the solution sufficient for
certain commercial authentication use-cases, but not all. The
averages shown here are the means over the individuals' FAR, FRR.




<Figure src="/deep-dives/pics/deep-dives/brain_based_auth_figure6.webp" alt="Summary of authentication performance in the field." caption={"<strong>Figure 6: Summary of authentication performance in the field.</strong> Forty nine participants were included in the test. Each participant had six genuine events, and 288 imposter events. All together in this test we used 294 genuine events, and 14112 imposter events. In A, and B histograms of authentication system performance at the participant level is present. In A, the performance regarding imposters (true rejection rate, and false acceptance rate). In B, the performance regarding genuines identities (true acceptance rate, and false rejection rate). These values were first calculated per each participant, and then distribution was calculated. C. A confusion matrix summary, showing the averaged performance over all participants. A detailed performance summary per each participant can be found in <strong>supp. Table 1</strong>."} />




Out of the total number of participants in this experiment (49), 37
participants have FRR=0, where 24 participants have FRR=0 and
FAR&lt;=9% (<strong>Supp. Table 1</strong>).




Next we asked what will be the performance of the authentication
system when only certain brain signal information is considered.
Explicitly, we repeated the training procedure (Methods) but this
time with only the power spectrum features of the following
brainwave modes: Delta(0.5-4Hz), Theta(4-8Hz), Alpha(8-12.5),
Beta(12.5-30Hz) and Gamma(30-48), and with some combinations
(Alpha-Beta, and Theta-Alpha-Beta). We found that usually for these
features, FRR can reach low values while the FAR is always kept high
(<strong>Table 1</strong>). As the number of features is increasing,
the better the performances of the authentication system. This
implies that more sophisticated models such as deep neural networks
will greatly improve the performance of the authentication system.
We will report on the results of different systems such as these in
future field test reports.




<Figure src="/deep-dives/pics/deep-dives/brain_based_auth_table1.webp" alt="Model performance as function of feature types." caption={"<strong>Table 1: Model performance as function of feature types. </strong>The same training and testing datasets were used for all models. The power spectrum density (PSD) of the following frequency bands were used as features. Delta(0.5-4Hz), Theta(4-8Hz), Alpha(8-12.5), Beta(12.5-30Hz) and Gamma(30-48). Each bandpass has four features, corresponding to the number of brain data channels. When using only one type of powerband feature, the averaged FRR may reach low levels, but the FAR always remains high."} />


<h2>Discussion</h2>


We performed a generalizable field test of a brain-based
authentication system that uses non-invasively measured brain
signals to verify user identity. All participants were completely
new (naïve users) to the system, enrolled themselves from home in a
self-guided tutorial, used a comfortable head wearable for hours at
a time without issue and performed repeated authentication measures
across multiple days. On the whole, this amounts to a reasonable
simulation of real contexts that enterprise and consumer
authentication methods need to operate in to be commercially viable.
Specifically, these methods must work regardless of time of day and
be robust to changes in brain state (pre/post-coffee, hunger,
wakefulness, awareness, etc.) and the ambient noise inherent to
measurements made outside of controlled laboratory conditions.




The main goal of this field test was to evaluate the base
feasibility of a scalable, commercial-grade brain ID authentication
system; advanced data engineering methods were not applied to boost
performance further. A simplified feature set and simple machine
learning methods were applied over a minimal training period of less
than three minutes enrollment data per participant. Amidst these
severe constraints on performance, brain-based authentication proved
to be approaching commercial-grade levels. In future tests the
parameters used will be optimized, here our authentication system
ran on suboptimal parameters that were fast calculated to serve as
more heuristics than anything. For example the epoch length, the
authentication event duration, selection of features by their
importance, or the amount of training data we know have an impact on
performance from previous and ongoing work. These are all tunable
parameters depending on the demands of the authentication task:
future research will clarify the timescales at which the optimal
information for identification verification occurs for each tier of
authentication system. We are confident that more sophisticated
machine-learning architectures together with other parameters
optimization will deliver superior, product level authentication
performance that will match or exceed the performance of top
non-brain authentication biometrics available today.




Wearables that touch the head, such as headphones or AR/VR, are a
natural form factor for brain-based authentication and we anticipate
that demand from enterprises and consumers will necessitate that
these devices evolve beyond passwords and fingerprints to iris ID
based on eye scanning and eventually brain ID, based on brain
scanning. The demand for both strong and convenient authentication
solutions for future head-worn devices drove our design of the
paradigm for prompt-response analysis here, and it is notable that
the rapid image prompt-response paradigm evaluated (with users
watching images on a tablet while wearing a headband) has been
validated by us elsewhere in AR (Supp. Video 2) using Microsoft
Hololens.




Given the performance obtained in this field test and the
ease-of-use of this method for head wearables, brain ID seems to be
one of the most intuitive and powerful authentication solutions for
next generation head-worn computers. Brain identities, like any
other biometric identity, will need to conform to privacy standards
and be offered within protected software and chip architectures such
as those pioneered for fingerprint scanners and face recognition,
but this is no limitation on the adoption of such a beneficial
method.




Biometrics as a class are uniquely comfortable and convenient to use
because they do not require the user to remember anything (like a
password), or carry anything (like a physical key). Biometrics
offset the cognitive load of password management plus the risks
associated with alphanumeric passwords, and even offer the promise
of obviating passwords altogether in future computing ecosystems.
For now, brain ID is at a nascent stage of industry adoption and the
solution presented here represents one of the more scalable designs,
since we can easily increase the dataset to more participants and
more events within the principled framework of forcing divergences
in inter-participant data and convergences in intra-participant
data.




Furthermore, the head wearable that people put on themselves in this
test to measure their brain signal is a consumer device that is
currently available worldwide, highlighting the lack of need for
exotic or rare materials to acquire sufficient brain signal to
measure brain IDs, nor the need for specialized laboratories or
facilities. More information in the brain signal remains unexplored
here, being outside the scope of the current field test and report.
Future research will develop concepts related to the theoretical and
practical information boundaries in brain signal, since for head
wearables in particular, brain biometric identity warrants continued
testing across expanded participant populations and implementation
in commercial devices that are optimized for given use-cases and
environments.


<h2>Conclusion</h2>


We showed that a relatively simple brain-based authentication system
can use non-invasively measured brain signals from consumer quality
head wearable devices to differentiate between users with a high
degree of certainty. Authentication using non-invasively-measured
brain signals in this way was found to not only be feasible, but
robust: the correlation matrices derived from the current test find
our computed brain identities to be readily distinguishable between
different participants and consistently similar among participants,
satisfying the core requirements of a commercial-grade biometric
authentication system. The complexity inherent to human brain
signals was, therefore, found to not be too volatile to be leveraged
for steady, reliable use as a passwordless authentication method.




We built and validated the method through a scalable software
infrastructure that was able to deliver brain-based authentication
at a commercial-grade, within a generalized framework that provides
for continual performance improvement with additions of new
participants. As the methods were designed around characteristic
patterns observable during limited windows of time, at any time, it
is clear that there is value to continued data collection at larger
scales and across additional contexts. For both inter-subject
variability and to further clarify the invariant patterns underlying
intra-participant variability, expanded data collection can be
beneficial. The present sample is sufficient however to conclude
that brain-based authentication is already a viable method for
certain commercial uses, and has the potential to serve many more in
the future.


<h2>Ethics declarations</h2>


All authors are employees of Arctop Inc.


<h2>Supplementary Materials</h2>


<Figure src="/deep-dives/pics/deep-dives/brain_based_auth_figure7.webp" alt="Brain biometric ID cluster tree." caption={"<strong>Supp. Figure 1: Brain biometric ID cluster tree.</strong> The mean overall training events were calculated for each participant. Mean event correlation matrix between participants was then calculated by pairwise correlation. Using this matrix, the hierarchical cluster tree (dendrogram) algorithm creates the linkage distance between participants (y-axis)."} />




<Figure src="/deep-dives/pics/deep-dives/brain_based_auth_figure8.webp" alt="Histograms of intra-participant and inter-participants events correlations." caption={"<strong>Supp. Figure 2: Histograms of intra-participant and inter-participants events correlations.</strong> Intra-participant events correlation is the mean of pairwise correlation between all training events of a participant with themselves. Inter-participant correlations are the mean of pairwise correlation of all training events of a participant with the events of another participant. The diagonal elements in the event correlation matrix (<strong>Figure 6E</strong>), represents the intra-participant correlations while the inter-participants correlations are the off-diagonal elements of the matrix. It is clearly seen that intra-participant correlations are generally higher than the inter-participants correlation. Meaning, a higher similarity within intra events patterns compared with inter-participants events. Although there is an overlap between the two histograms, it does not necessarily mean that perfect separability at the authentication event level is not feasible, as suggested by the inset."} />




<Figure src="/deep-dives/pics/deep-dives/brain_based_auth_figure9.webp" alt="Epoch predictions and threshold determination." caption={"<strong>Supp. Figure 3: Epoch predictions and threshold determination.</strong> Three sessions per participant are contributing to the training data. Out of it, 30% are devoted for model validation, and to determine the model threshold. In panels A, B, C the prediction of three models trained for three participants (sub #23, # 35, #34 respectively) are presented for the validation data. Here, epochs predictions of genuine identity are marked in black dots, and epochs predictions of imposters are marked in red. The threshold (black dashed line), discernmenting between genuine and imposter epochs is determined by an optimization algorithm. The algorithm finds a threshold probability in which the false acceptance rate (FAR) is minimal while the true acceptance rate (1-FRR) is maximal. This is under the condition for TAR&gt;90%, and FAR&lt;3%. This is demonstrated in panels A1, B1, C1. FAR, TAR functions are plotted in red and black respectively, the threshold which was found is marked in black dashed-dot line, y-axis is in logarithmic scale."} />




<Figure src="/deep-dives/pics/deep-dives/brain_based_auth_figure10.webp" alt="Event prediction." caption={"<strong>Supp. Figure 4: Event prediction.</strong> The events are segmented into 31 epochs. The probability of each epoch to be of a genuine identity or of an imposter one is determined by a model. Probabilities above the threshold (black dash line) belong to genuine identity, and if below the threshold, to an imposter. Threshold was determined previously in the training process (see <strong>Supp. Figure 3</strong>). A, B, C the predictions of three models trained for three participants (sub #23, # 35, #34 respectively) are presented. Here the epochs under test are only of genuine identity (black dots). While in A, and C all predictions are above threshold, in B some of the predictions are below the threshold. The final decision whether the event is of genuine identity is determined only if more than 40% of epochs are above the threshold. In A1, B1, C1 full test prediction is shown for the same three participants. The test included 294 events, where each participant has 6 genuine events. Events of genuine identity are marked in black dots and imposter events are in red dots. In all three cases all genuine events were identified correctly, having zero false rejection rate (FRR=0). As for the imposters, only in C1, all imposter events are below the threshold, with zero false acceptance rate (FAR=0). The final FRR and FAR of each participant is shown in the yellow windows."} />




<Figure src="/deep-dives/pics/deep-dives/brain_based_auth_table2.webp" alt="A detailed performance of the authentication system for each participant." caption={"<strong>Supp. Table 1: A detailed performance of the authentication system for each participant. </strong>The coefficients of the confusion matrix per each participant is presented."} />




<strong>Supp. Video 1: Rapid Serial Visual Presentation (RSVP) Stimuli.</strong> Example of a stream of images watched by participants while brain signals
were recorded by their headband. <a href="https://youtu.be/TWUzbX3Q8sk">https://youtu.be/TWUzbX3Q8sk</a>




<strong>Supp. Video 2: Brain-based Authentication: Living Room Demo.</strong> Microsoft HoloLens 1, retrofitted with BCI sensors, delivers passwordless
authentication. <a href="https://youtu.be/n6v9z3lNs2M">https://youtu.be/n6v9z3lNs2M</a>





<References items={[{"html":"Stassen, H. H., G. Bomben, and P. Propping. \"Genetic aspects of the EEG: an investigation into the within-pair similarity of monozygotic and dizygotic twins with a new method of analysis.\" <em>Electroencephalography and clinical neurophysiology</em> 66, no. 6 (1987): 489–501."},{"html":"Poulos, Marios, Maria Rangoussi, and Nikolaos Alexandris. \"Neural network based person identification using EEG features.\" In 1999 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings. ICASSP99 (Cat. No. 99CH36258), vol. 2, pp. 1117–1120. IEEE, 1999."},{"html":"Poulos, M., M. Rangoussi, N. Alexandris, and A. Evangelou. \"Person identification from the EEG using nonlinear signal classification.\" <em>Methods of information in Medicine</em> 41, no. 1 (2002): 64–75.<a href=\"https://www.biorxiv.org/lookup/external-ref?access_num=11933767&link_type=MED&atom=%2Fbiorxiv%2Fearly%2F2021%2F04%2F30%2F2021.04.09.439244.atom\"></a>"},{"html":"Marcel, Sebastien, and José del R. Millán. \"Person authentication using brainwaves (EEG) and maximum a posteriori model adaptation.\" <em>IEEE transactions on pattern analysis and machine intelligence</em> 29, no. 4 (2007): 743–752."},{"html":"Van Beijsterveldt, C. E. M., and G. C. M. Van Baal. \"Twin and family studies of the human electroencephalogram: a review and a meta-analysis.\" <em>Biological psychology</em> 61, no. 1-2 (2002): 111–138. <a href=\"https://www.biorxiv.org/lookup/external-ref?access_num=10.1016/S0301-0511(02)00055-8&link_type=DOI\"></a>"},{"html":"Thorpe, Julie, Paul C. Van Oorschot, and Anil Somayaji. \"Pass-thoughts: authenticating with our minds.\" <em>In Proceedings of the 2005 workshop on New security paradigms</em>, pp. 45–56. 2005."},{"html":"Wang, Min, Jiankun Hu, and Hussein A. Abbass. \"BrainPrint: EEG biometric identification based on analyzing brain connectivity graphs.\" <em>Pattern Recognition</em> 105 (2020): 107381."},{"html":"Piplani, Tanya, Nick Merill, and John Chuang. \"Faking it, Making it: Fooling and Improving Brain-Based Authentication with Generative Adversarial Networks.\" In 2018 IEEE 9th International Conference on Biometrics Theory, Applications and Systems (BTAS), pp. 1–7. IEEE, 2018."},{"html":"Abo-Zahhad, Mohammed, Sabah Mohammed Ahmed, and Sherif Nagib Abbas. \"State-of-the-art methods and future perspectives for personal recognition based on electroencephalogram signals.\" <em>IET Biometrics</em> 4, no. 3 (2015): 179–190."},{"html":"Armstrong, Blair C., Maria V. Ruiz-Blondet, Negin Khalifian, Kenneth J. Kurtz, Zhanpeng Jin, and Sarah Laszlo. \"Brainprint: Assessing the uniqueness, collectability, and permanence of a novel method for ERP biometrics.\" <em>Neurocomputing</em> 166 (2015): 59–67."},{"html":"Ashby, Corey, Amit Bhatia, Francesco Tenore, and Jacob Vogelstein. \"Low-cost electroencephalogram (EEG) based authentication.\" In 2011 5th International IEEE/EMBS Conference on Neural Engineering, pp. 442–445. IEEE, 2011."},{"html":"Campisi, Patrizio, and Daria La Rocca. \"Brain waves for automatic biometric-based user recognition.\" <em>IEEE transactions on information forensics and security</em> 9, no. 5 (2014): 782–800."},{"html":"Maiorana, Emanuele, Daria La Rocca, and Patrizio Campisi. \"On the permanence of EEG signals for biometric recognition.\" <em>IEEE Transactions on Information Forensics and Security</em> 11, no. 1 (2015): 163–175."},{"html":"Mohanchandra, Kusuma, G. M. Lingaraju, Prashanth Kambli, and Vinay Krishnamurthy. \"Using brain waves as new biometric feature for authenticating a computer user in real-time.\" <em>International Journal of Biometrics and Bioinformatics (IJBB)</em> 7, no. 1 (2013): 49."},{"html":"Mu, Zhendong, Jianfeng Hu, and Jianliang Min. \"EEG-based person authentication using a fuzzy entropy-related approach with two electrodes.\" <em>Entropy</em> 18, no. 12 (2016): 432."},{"html":"Palaniappan, Ramaswamy. \"Two-stage biometric authentication method using thought activity brain waves.\" <em>International journal of neural systems</em> 18, no. 01 (2008): 59–66."},{"html":"Paranjape, R. B., J. Mahovsky, L. Benedicenti, and Z. Koles. \"The electroencephalogram as a biometric.\" In Canadian Conference on Electrical and Computer Engineering 2001. Conference Proceedings (Cat. No. 01TH8555), vol. 2, pp. 1363–1366. IEEE, 2001."},{"html":"Ruiz-Blondet, Maria V., Zhanpeng Jin, and Sarah Laszlo. \"CEREBRE: A novel method for very high accuracy event-related potential biometric identification.\" <em>IEEE Transactions on Information Forensics and Security</em> 11, no. 7 (2016): 1618–1629."},{"html":"Thomas, Kavitha P., and A. Prasad Vinod. \"EEG-based biometric authentication using gamma band power during rest state.\" <em>Circuits, Systems, and Signal Processing</em> 37, no. 1 (2018): 277–289."},{"html":"Yang, Su, and Farzin Deravi. \"On the usability of electroencephalographic signals for biometric recognition: A survey.\" <em>IEEE Transactions on Human-Machine Systems</em> 47, no. 6 (2017): 958–969."},{"html":"Revett, Kenneth, and Sergio Tenreiro de Magalhães. \"Cognitive biometrics: Challenges for the future.\" <em>In International Conference on Global Security, Safety, and Sustainability</em>, pp. 79–86. Springer, Berlin, Heidelberg, 2010."},{"html":"Curran, Max T., Jong-kai Yang, Nick Merrill, and John Chuang. \"Passthoughts authentication with low cost EarEEG.\" In 2016 38th Annual international conference of the IEEE engineering in medicine and biology society (EMBC), pp. 1979–1982. IEEE, 2016"},{"html":"Del Pozo-Banos, Marcos, Jesús B. Alonso, Jaime R. Ticay-Rivas, and Carlos M. Travieso. \"Electroencephalogram subject identification: A review.\" <em>Expert Systems with Applications</em> 41, no. 15 (2014): 6537–6554."},{"html":"Marcel, Sebastien, and José del R. Millán. \"Person authentication using brainwaves (EEG) and maximum a posteriori model adaptation.\" <em>IEEE transactions on pattern analysis and machine intelligence</em> 29, no. 4 (2007): 743–752."},{"html":"Näpflin, Markus, Marc Wildi, and Johannes Sarnthein. \"Test–retest reliability of resting EEG spectra validates a statistical signature of persons.\" <em>Clinical Neurophysiology</em> 118, no. 11 (2007): 2519–2524."},{"html":"Näpflin, Markus, Marc Wildi, and Johannes Sarnthein. \"Test–retest reliability of EEG spectra during a working memory task.\" <em>Neuroimage</em> 43, no. 4 (2008): 687–693."},{"html":"Haruvi Aia, Ronen Kopito, Noa Brande-Eilat, Shai Kalev, Eitan Kay, and Dan Furman. \"Differences in the effects on human focus of music playlists and personalized soundscapes, as measured by brain signals.\" <em>Biorxiv</em> (2021)."}]} />




</div>

---

# Modeling The Effect of Background Sounds on Human Focus Using Brain Decoding Technology
URL: https://arctop.com/deep-dives/modeling-focus
Published: 2021-04-02

This paper is available in PDF format <a href="https://www.biorxiv.org/content/10.1101/2021.04.02.438269v3.full.pdf">here</a>.


<h2>Abstract</h2>


The goal of this study was to investigate the effect of sounds on human
focus and to identify the properties that contribute most to increasing
and decreasing focus in people within their natural, everyday
environment. Participants (N=62, 18-65y) performed various tasks on a
tablet computer while listening to either no background sounds
(silence), popular music playlists designed to increase focus
(pre-recorded songs in a particular sequence), or engineered soundscapes
that were personalized to individual listeners (digital audio composed
in real-time based on input parameters such as heart rate, time of day,
location, etc.). Sounds were delivered to participants through
headphones while simultaneously their brain signals were recorded by a
portable electroencephalography headband. Participants completed four
one-hour long sessions at home during which different sound content
played continuously. Using brain decoding technology, we obtained
individual participant focus levels over time and used this data to
analyze the effects of various properties of sound. We found that while
participants were working, personalized soundscapes increased their
focus significantly above silence (p=0.008), while music playlists did
not have a significant effect. For the young adult demographic (18-36y),
all sound content tested was significantly better than silence at
producing focus (p=0.001-0.009). Personalized soundscapes increased
focus the most relative to silence, but playlists of pre-recorded songs
also increased focus significantly during specific time intervals.
Ultimately we found that it is possible to accurately predict human
focus levels that will be experienced in response to sounds <em>a priori</em> based on the sound's physical properties. We then applied this finding
to compare between music genres and revealed that classical music, engineered
soundscapes, and natural sounds were the best genres for increasing focus,
while pop and hip-hop were the worst. These insights can enable human and
artificial intelligence composers to produce increases or decreases in listener
focus with high temporal (millisecond) precision. Future research will include
real-time adaptation of sound libraries for other functional objectives beyond
affecting focus, such as affecting listener enjoyment, stress, and memory.


<h2>Introduction</h2>
<h3>The effect of sound on human experience</h3>


Sounds are all around us, from natural sounds like the wind, to
engineered sounds like music. It is well-established that sounds have a
major influence on the human brain and consequently, human experience (<strong>Levitin, 2006</strong>; <strong>Sacks, 2010</strong>). Sounds can reduce stress (<strong>Davis &amp; Thaut, 1989</strong>), support learning and memory formation (<strong>Hallam et al., 2002</strong>), improve mood (<strong>Chanda &amp; Levitin, 2013</strong>), and
increase motivation (<strong>Salimpoor et al., 2015</strong>). Sounds
can also do the opposite and create aversive experiences (<strong>Kumar et al., 2012</strong>; <strong>Schreiber &amp; Kahneman, 2000</strong>; <strong>Zald &amp; Pardo, 2002</strong>). One of the most significant effects of sounds is to impact focus.
Focus is commonly demanded by tasks of daily living and work, and in
these areas sounds offer a safe way to increase focus levels and
productivity. However, sounds can be beneficial or distracting, and
previous results have been inconclusive in determining the reason (<strong>de la Mora Velasco &amp; Hirumi, 2020</strong>).




For example, it has been found that listening to music with lyrics while
reading or working can decrease concentration or cognitive performance
(H. <strong>Liu et al., 2021</strong>; <strong>Shih et al., 2012</strong>), while several studies have shown oppositely that natural-occurring
sounds such as white noise, as well as classical music, can be
beneficial for increasing focus and can improve learning outcomes (<strong>Angwin et al., 2017</strong>; <strong>Chou, 2010</strong>; <strong>Davies, 2000</strong>; <strong>Gao et al., 2020</strong>). Therefore, one interesting question emerges which is: what are the
specific properties of sounds that affect human focus levels the most?
Additionally, studies have shown that the effect of sounds is often
subjective, where whether one likes a given sound or not is a key factor
in its effect on their experience (<strong>Cassidy &amp; Macdonald, 2009</strong>; <strong>Huang &amp; Shih, 2011</strong>; <strong>Mori et al., 2014</strong>). Although this finding about the subjectivity of sound reappears
across many studies, psychophysical thresholds are known to exist and
there are clearly natural laws governing much of the way humans hear and
experience sound (<strong>Levitin et al., 2012</strong>; <strong>Nia et al., 2015</strong>; <strong>Washburne, 2020</strong>).




The potential of sounds to increase focus and demand for
non-pharmaceutical tools that enable individuals to enhance their
ability to focus has recently led several companies (including Endel,
Brain.fm, Mubert, Enophone, Focus@Will, Melodia, AIVA, and others) to
develop soundscapes that are dedicated to increasing focus on-demand.
These soundscapes include elements of white noise, music, and other
sonic properties that are functionally combined to increase a listener's
focus and maintain high levels of focus over long durations of time. One
of the challenges in this field is to figure out the physical properties
of sound that contribute to human experience the most so that design
principles can be defined correctly to create soundscapes that achieve
the goal of increasing focus, opposed to the inverse of causing
distractions and hurting an individual's ability to focus. Insights
about sound properties therefore have been sought by commercial groups
alongside academic groups in order to learn how to optimize experiences
through sound.




Many scientific studies have explored this question and looked for the
relationship between sound, music and human experience using objective
measures that empirically assess properties of audio and their emotional
correlates. For example, Cheung et al (<strong>Cheung et al., 2019</strong>) found that pleasure from music depends on states of expectation, such
as a skipped rhythmic beat, which can either be pleasurable or
discomforting depending on the listener's circumstance. <em>Sweet Anticipation</em> (<strong>Huron, 2006</strong>) similarly maps how music evokes
emotions within a theory of expectation and describes psychological
mechanisms that are responsible for many people's mixed responses to
sounds. Other studies used machine learning methods to map from features
of audio signals to emotions (<strong>Brotzer et al., 2019</strong>; <strong>Cunningham et al., 2020</strong>; <strong>Hizlisoy et al., 2021</strong>; <strong>Vempala &amp; Russo, 2012</strong>; <strong>Yang et al., 2008</strong>). These machine learning studies
to date have, however, only aimed to predict emotions based on the
limited valence-arousal circumplex model, and as far as we know, no
attempts have been made to predict human focus levels exclusively based
on audio signal analysis.




One persistent obstacle to the field's understanding has been studies
that rely on data with a low temporal resolution. Since sounds and
emotions evolve fast, on the order of tens of milliseconds, the current
lack of modeling tools capable of capturing the fast changes in human
experience that accompany changes in sound is a major hindrance to
progress (<strong>Cowen &amp; Keltner, 2017</strong>; <strong>Larsen &amp; Diener, 1992</strong>). Commonly, for example, reports are based on data where there is a
single emotional label per song, while the song lasts ~2-3 minutes and
throughout it there are emotional dynamics that change dramatically.
This mismatch of data can lead to conclusions being drawn from
inadequately small amounts of samples, and worse than that, inaccurate
emotional labels.


<h3>Attention and emotion decoding from brain signal</h3>


Brain decoding technology offers an exceptional opportunity to tackle
this issue, since it enables us to get an estimation for the experience
dynamics at the same time resolution as focus phenomena occur. Using
electroencephalogram (EEG) sensor data, which contains electrical brain
activity measured from the scalp (non-invasive) on the order of hundreds
of measurements per second, many studies have established that it is
possible to capture fast changes in human emotions and experience, such
as stress (<strong>Perez-Valero et al., 2021</strong>), arousal (<strong>Faller et al., 2019</strong>), fatigue (<strong>Hu, 2017</strong>), and happiness (<strong>Lin et al., 2017</strong>). Several studies have similarly shown the ability to capture focus
and attentional state changes, affirming that this information as well
is captured in EEG sensor data (<strong>Hamadicharef et al., 2009</strong>; <strong>Jung et al., 1997</strong>; <strong>Micoulaud-Franchi et al., 2014</strong>; <strong>Tuckute et al., 2021</strong>). While brain decoding
technology has been applied widely to study the effects of different
types of stimuli (e.g visual, tactile, auditory) on human experience (<strong>Asif et al., 2019</strong>; <strong>Bhatti et al., 2016</strong>; <strong>Shahabi &amp; Moghimi, 2016</strong>), as far as we know, it has not been applied to study the joint
effects of sound and focus at the high temporal resolution needed to
explain both phenomena.




In recent years, progress in the development of non-clinical, wearable
EEG sensors (such as Muse, Neurosky, Emotiv, Bitbrain, etc.), which are
intended for consumer uses, has led to new research paradigms where
comfortable, affordable, wireless, and easy-to-use at-home measurement
devices collect neuroscientific data "in-the-wild" at a large scale and
make it possible for the first time to measure brain responses from
diverse audiences within their natural, real-world environment. Many of
the wearable devices offer decoding outputs beyond the raw sensor data,
and these "off-the-shelf" decoding outputs include attention,
relaxation, and other states (<strong>Abiri et al., 2019</strong>; <strong>Bird et al., 2019</strong>; <strong>González et al., 2015</strong>; <strong>Liu et al., 2013</strong>; <strong>Rebolledo-Mendez et al., 2009</strong>). It is important to
note, however, that although decoder algorithms exist in the market for
consumer uses, verifying their reliability to accurately capture
attention, valence, arousal, stress and other attributes of human
experience at a high temporal resolution and research quality has
remained a challenge.


<h3>Combining brain decoding with sound tests to increase focus</h3>


In the current study, we used Arctop's brain decoding technology (neuOS<sup>TM</sup>) on data from portable EEG (Muse-S) headbands to measure human focus
levels in individuals performing tasks at home while listening to
different types of sounds. Since this is a relatively new decoding
technology, we first evaluate the validity of the focus outputs within
the experimental conditions. Then, once convinced of the output's
veracity and reliability, we use the focus data to compare effects of
different sound stimuli on individuals while performing different tasks.




Next, we exploit the decoded data's high time resolution to map between
raw audio signals and the focus dynamics. Based on this mapping, we
build a model that takes sound properties and predicts human focus
levels, enabling us to compare between new songs, sounds, and between
genres to gain additional insights about the nature of sounds which
drives human focus the most. These insights can help in the future to
generate optimal playlists to increase focus, engineer better
soundscapes, and even adapt sounds in real-time based on an individual's
focus levels to enable them to precisely influence their own mental
state.


<h2>Materials and Methods</h2>
<h3>Participants</h3>


Sixty-two (62) participants (40 males, 22 females, 18-65 years),
completed four (4) sessions over a single (1) week at their own home.
All participants were recruited from an opt-in screening panel and were
distributed across the five (5) major regions of the continental United
States (Northeast, Southwest, West, Southeast, and Midwest). Only
participants who reported normal hearing, normal vision, or vision that
was corrected to normal with contact lenses, were included. We excluded
volunteers who reported using medication that might influence the
experiment and who reported neurological or psychiatric conditions that
could influence the results. Participants were native English speakers
and a written informed consent was obtained from each participant prior
to their participation. Participants received compensation for their
time.


<h3>Paradigm</h3>
<h4>Tasks</h4>


Participants performed various tasks within a mobile app (neuOS<sup>TM</sup> by Arctop Inc.) while listening to one of three types of sound and wearing
a brain signal measuring headband (4-channel EEG Muse-S device by Interaxon
Inc.). Each participant received a kit at their home that included all the
equipment needed to participate, including over-ear (Sony) headphones, headband
and tablet computer with the mobile app installed. Participants recorded
four one hour long sessions, while listening to different types of sounds.
Sessions included 30 minutes of a "Preferred Task" — a task chosen by the
participant — followed by short tasks used to validate the brain decoding
outputs for each session. These validation tasks included video games (Tetris),
math problems (Arithmetics), and word problems (Creativity). Participants
were assigned to groups according to a pseudorandom schedule that controlled
for potential sequence effects of the tasks and different sound stimulus
types (<strong>Fig. 1</strong>). The short tasks were used to calibrate
the Arctop decoding algorithms to a validated performance level, and
afterwards the validated model was used to measure each participant's
focus level across the Preferred Task.




<Figure src="/deep-dives/pics/deep-dives/modeling_focus_figure1.webp" alt="Schematic illustration of the paradigm in each recording session." caption={"<strong>Figure 1. Schematic illustration of the paradigm in each recording session.</strong> <br />Each session started with 30 minutes of a task selected by the participant (\"Preferred Task\"), followed by 3 minutes of arithmetics exercises, 3 minutes of a creativity task, and two levels of Tetris the video game (each level lasted 1 minute regardless of performance). After each task, participants answered a survey where they reported on aspects of their experience (e.g. focus, enjoyment, stress) using linear scale sliders from \"Not at all\" (0) to \"Very\" (1)."} />




Participants were instructed to choose a Preferred Task they could
perform in a seated position while listening to sounds through the
headphones, and which they would be happy to repeat in all four
sessions. For example, Preferred Tasks that were chosen included
knitting, working, reading, and solving Sudoku puzzles. At the end of
each task the participants self-reported their experience through a
survey in the app which used linearly-scaled slider buttons to quantify
experience along several dimensions (e.g. focus level, enjoyment,
stress, motivation, etc.). For the Preferred Task, the survey included
reporting on their focus level during the first and second half of the
task separately, resulting in six (6) self-reported quantitative focus
labels per session (Preferred Task: 2 labels, arithmetics: 1 label,
creativity: 1 label, tetris: 2 labels).


<h4>Sounds</h4>


Each participant experienced four sound conditions over the four days of
the study: two music playlists by leading digital service providers
Spotify and Apple (downloaded September 2020), one personalized
soundscape engineered by Endel, and silence (no audible sounds). We
selected Spotify's 'Focus Flow' playlist and Apple Music's 'Pure Focus'
playlist to represent the category of pre-recorded sounds designed to
increase focus. For soundscapes we selected the mobile application Endel
to represent the category of real-time, engineered sounds that contain a
mixture of noise and musical properties. The Endel app 'Focus'
soundscape was used by each participant on their own device. All sound
conditions were instrumental (i.e. did not include singing or any
audible lyrics). For the condition of silence, participants wore
headphones exactly as they did in the sound conditions, but no music or
audible sounds of any kind were played and no soundscape was generated -
participants simply completed the session in a quiet environment.


<h3>Data processing</h3>
<h4>Data acquisition</h4>


While participants were listening to sounds and engaging in the
experimental tasks, their electrical brain activity was recorded using a
portable, noninvasive electroencephalograph (EEG) headband that weighed
41 grams (Muse-S device by Interaxon Inc). The headband included four
dry fabric EEG sensors (sampling rate: 256 Hz), photoplethysmography
(PPG) sensors (for heart rate) and motion sensors (gyroscope,
accelerometer). The brain-measuring EEG sensors are located on the scalp
at two frontal channels (AF7, AF8) and two temporal channels (TP9,
TP10), with the reference channel at Fpz. The headbands were put on by
participants themselves with the assistance of a quality control screen
that started each session by giving participants real-time feedback on
the signal quality and made it easy to adjust the headband appropriately
to acquire an optimal signal quality (<strong>Fig. 2</strong>). No
technicians or other support staff assisted in the placement of the
headbands - the process was completely automated by the in-app prompts,
freeing the participants to complete sessions at any time of their
choosing.




<Figure src="/deep-dives/pics/deep-dives/modeling_focus_figure2.webp" alt="Schematic illustration of the data processing pipeline." caption={"<strong>Figure 2. Schematic illustration of the data processing pipeline. </strong><br />Data acquisition included at-home recordings of four sessions, each with a different background sound type. Arctop's neuOS brain decoding technology was used to predict the focus dynamics at a rate of 5Hz. Obtaining the brain decoded focus dynamics synchronously with the sound content enables comparison of focus levels correlated with different physical properties of sound."} />


<h4>Brain data based models of focus</h4>


Arctop brain decoding technology (neuOS) was used to transform the
sensor data into predicted focus dynamics with a time resolution of 5Hz
(<strong>Fig 2</strong>). For each participant, short tasks (games, word
and math problems) were used to calibrate and validate a model of their
focus based on the brain data, and then once validated the model was
applied to the Preferred Task data. <strong>Fig. 3</strong> shows the resulting
brain decoded focus levels of two representative participants across all
four sessions during the Preferred Task. Model performance was evaluated
using Pearson correlation coefficient between the self-reported focus and
the brain decoded focus values, and after thresholding the values, with the
area under the ROC curve for binary classification of low/high focus (<strong>Fig. 5</strong>). Eleven (11) participants were excluded from further analysis
following model validation due to excessive noise in their recorded
brain data and/or unreliable survey responses, leaving a total of 51
participants (mean age= 36, SD=8, 17 females and 34 males) in the
experimental analysis.




<Figure src="/deep-dives/pics/deep-dives/modeling_focus_figure3.webp" alt="Brain data based focus model dynamics of two representative participants during the Preferred Task performed at each of the four sessions." caption={"<strong>Figure 3. Brain data based focus model dynamics of two representative participants during the Preferred Task performed at each of the four sessions.</strong> Each row represents a session with a different sound stream playing in the background as participants perform their chosen task. Each session included 30 minutes (X axis = time in minutes) of a \"Preferred Task\" over which their focus level (Y axis = decoded focus) was measured. Participant 29 <strong>(A)</strong> was reading while Participant 45 <strong>(B)</strong> was working."} />


<h4>Statistical methods</h4>


For comparisons between average focus levels during the different sound
content presented, we calculated for each participant (N=51) the median
focus level while performing the preferred task and conducted a one-way
repeated measures ANOVA (Analysis of variance) test. Then, if p&lt;0.05,
paired t-tests were applied post hoc to compare between pairs of sound
streams using Holm-Bonferroni correction. Time series statistical tests
were applied to compare focus level dynamics and discover specific time
periods of significant difference. A paired t-test was applied at each
second between focus levels of two sound streams. The p-values were then
corrected for multiple comparisons by setting a threshold for a minimum
significant sequential time-samples. The threshold was determined by
random permutations (1000 iterations) of participants' conditions and
repeating the statistical test, resulting in a distribution of
significant sequential time samples. The threshold was set as the 95%
percentile of the resultant distribution (<strong>Broday-Dvir et al., 2018</strong>).


<h4>Sound signal decomposition and feature extraction</h4>


The pre-recorded playlists conditions (Apple and Spotify) provided raw
sound data that we used to obtain sound property dynamics in the time
and frequency domain that could be correlated with the obtained focus
dynamics. Soundscapes were not used in this analysis because they were
produced in real-time personally for each participant, which limited the
ability to apply sound property analysis appropriately across the data
set. The sound features were calculated using Python's library
pyAudioAnalysis (<strong>Giannakopoulos, 2015</strong>), for example,
the sound signal energy, spectral entropy, and chroma coefficients. The
features were calculated in short-time windows of 50 ms with a sliding
window of 25 ms. Then, basic statistics were calculated over the sound
features in windows of 30 seconds (e.g. mean and std), resulting in 136
sound properties (<a href="https://github.com/tyiannak/pyAudioAnalysis/wiki/3.-Feature-Extraction">link to full list</a>). To enable mapping to the brain model, the brain decoded focus levels
were also averaged in the corresponding 30 seconds windows (<strong>Fig. 4</strong>).




<Figure src="/deep-dives/pics/deep-dives/modeling_focus_figure4.webp" alt="Diagram demonstrating the framework for correlation of time-series focus values with sound properties." caption={"<strong>Figure 4. Diagram demonstrating the framework for correlation of time-series focus values with sound properties. (A)</strong> Example of a recorded brain data in microvolts (single channel of EEG) segment, which after applying the preprocessing and trained models on 30 minutes of recordings, transforms to the brain decoded focus dynamics (<strong>top (C)</strong>). <strong>(B)</strong> Examples of a sound segment in decibels taken from one of the songs. <strong>(C)</strong> The sound features (y-axis) dynamics during 30 minutes of recordings (x-axis)."} />




To obtain the threshold for significant correlations between sound
features and focus levels (p&lt;0.05), a shuffle analysis was performed.
Random permutations (1000 iterations) of the brain decoded focus levels
were applied across songs to preserve the time dependency of focus
levels within a song and the focus levels distribution. The correlation
of each sound feature was calculated with the permuted focus level. The
threshold was set as the 95% percentile of the resulting correlation's
distribution.


<h4>Obtaining the sound decoded focus model</h4>


To map the relationship between properties of the sounds heard and focus
levels measured from the brain, we first applied principal component
analysis (PCA) to reduce the dimensionality of the sound features. We
then trained regression models between the transformed sound features
and the brain decoded focus through a 5-fold cross validation procedure
that used 80% of the songs in each iteration to train and 20% to test.
The presented sound decoded focus model is a linear model based on the
first PCA component of the features (shifted and rescaled).


<h2>Results</h2>
<h3>Brain-measured focus levels accurately reflect self-reported focus levels</h3>


Before comparing focus levels elicited by the different sound types, we
validated the underlying brain decoding technology by comparing between
the brain-based focus predictions and the self-reported focus values. <strong>Figure 5A</strong> shows a histogram of the model performance per participant. The model is
evaluated based on the AUC score (of the ROC curve) for prediction of self-reported
focus during the Preferred Task (low-high focus) where the chance guessing
level is = 0.5 (black dashed line). The average result across participants
obtained was &lt;AUC&gt;=0.83 (N=51, SD=0.19), a strong validation of the
brain-measured focus model's accuracy.




<Figure src="/deep-dives/pics/deep-dives/modeling_focus_figure5.webp" alt="Validation of focus measurements derived from brain data." caption={"<strong>Figure 5. Validation of focus measurements derived from brain data.</strong><br /><strong>(A)</strong> Histogram of focus models performance per participant (N=51), evaluated using the area under the ROC curve (AUC-ROC). Black dashed line marks chance level (0.5). <strong>(B)</strong> Average focus levels per event vs. self-reported focus resulted in Pearson correlation of 0.6. Inset shows ROC curves for different values of self-report threshold. <strong>(C)</strong> Confusion matrix after thresholding the focus score predictions and self-report. Classification scores for 2-classes (low focus vs. high focus) are AUC=0.87 (area under ROC curve), Accuracy=0.8. <strong>(D)</strong> Average brain decoded focus levels vs. average self-reported focus across the four sound types."} />




When aggregating the tasks from all participants, the Pearson
correlation between the brain decoded focus model and the self-reported
focus was Corr(414)=0.6, p&lt;10<sup>−4</sup> (<strong>Fig. 5B</strong>). The inset in <strong>Figure 5B</strong> shows the ROC curves for different
values of self-reported threshold and the confusion matrix for one of these
thresholds (0.4) resulted in an accuracy score of 0.8 (<strong>Fig. 5C</strong>). <strong>Figure 5D</strong> shows the average brain decoded focus level
per sound type vs. the average self-reported score.


<h3>Soundscapes induce a higher focus level compared to silence</h3>


Using the validated focus models which output five measurements per
second (5Hz), we then compared between the average focus levels elicited
by the sounds during the Preferred Task. The background sound was found
to have a significant effect (top row in <strong>Table 1</strong>,
F(3,150)=4.28, p=0.006, statistical methods for details) on the elicited
focus level, and the post hoc tests (Holm-Bonferroni correction)
revealed that streaming soundscapes (with Endel app) was significantly
higher compared to silence (<strong>Fig. 6A1</strong>, supp. Table 1,
M=0.090, SE=0.027, t(50)=-3.38, p=0.008), while streaming music using
Apple or Spotify did not have an effect (Apple: t(50)=-2.37, p=0.11,
Spotify: t(50)=-1.24, p=0.65). For 35.3% of the participants the Endel
session produced their highest focus level, while for 27.5% of
participants the Apple playlist produced the highest focus level. For
19.6% of participants Spotify was best for producing focus and for 17.6%
silence was (<strong>Fig. 6A2</strong>, the details sorted focus levels
per participant are shown in Supp. Fig. 1).




<Figure src="/deep-dives/pics/deep-dives/modeling_focus_table1.webp" alt="Results of a one-way repeated measures ANOVA performed on each subgroup comparing the average brain decoded focus levels of each sound stream during the Preferred Task." caption={"<strong>Table 1. Results of a one-way repeated measures ANOVA performed on each subgroup </strong>comparing the average brain decoded focus levels of each sound stream during the Preferred Task. Sound most significantly affected those below 36 years old."} />




<Figure src="/deep-dives/pics/deep-dives/modeling_focus_figure6.webp" alt="Comparison of the brain decoded focus during the Preferred Task while listening to different sounds." caption={"<strong>Figure 6. Comparison of the brain decoded focus during the Preferred Task while listening to different sounds.</strong><br />Top row - Average focus levels for each sound stream during the Preferred Task for each group of interest, including statistical results. Error bars are standard errors. Bottom row - Distribution of the best session (highest focus on average) for each participant per group. The groups of interest are: <strong>(A)</strong> All participants (51), <strong>(B)</strong> Participants who were working during the Preferred Task (26), <strong>(C)</strong> Participants who were not working (25 -reading, knitting, playing, etc). <strong>(D)</strong> Participants above 36 (26). <strong>(E)</strong> Participants below 36 (25)."} />




To gain a better understanding of the conditions where sound affected
focus, we next split the participants into subgroups of interest and
repeated the statistical analysis. We first asked whether the focus
level difference is task dependent. During the Preferred Task, 51% of
the participants (26) chose to work, while the rest (49%) read a book
(29.4%), played games (9.8%) or did other various tasks (e.g. knitting,
9.8%). To assess the effect of sounds on focus levels during these
different tasks, we split the participants to the ones who worked and
those that did other tasks. We found that for the "working" group, the
focus level elicited by Endel's soundscapes was higher compared to
silence (M=0.12, SE=0.04, t(25)=3.26, p=0.017), while for the
"not-working" group there was no difference (<strong>Fig. 6</strong>,
supp. Table 1, M=0.06, SE=0.04, t(24)=1.552, p=0.447). These results
suggest that the focus level differences between Endel and Silence are
task-dependent, where the sound is beneficial for specific types of
tasks, namely, "working."




We next split the participants into two age groups according to the
median age (36 years). We found that for the younger participants
(age&lt;36, N=25), all sound types were superior to silence for
producing elevated focus levels (<strong>Fig. 6</strong>, supp. Table 1,
M=0.14,0.13,0.12, SE=0.04,0.03,0.03, t(24)=3.79,4.49,3.67,
p=0.004,0.001,0.005 for Endel, Apple and Spotify respectively) while for
the older participants (age&gt;36, N=26), there was no difference
between sound and silence. The focus level differences were therefore
found to also be age-dependent.


<h3>Time series analysis of the focus dynamics reveal differences between all sound types and silence</h3>


Exploiting the high temporal resolution of the focus measurements, we
compared the focus dynamics to each sound stream that played during the
30 minutes of the Preferred Task (<strong>Fig. 7</strong>, <strong>table 2</strong>). When comparing Endel's soundscapes vs. Silence (<strong>Fig. 7A</strong>), we found that the focus level elicited by Endel's soundscape was
higher 87% of the time, a separation whose significance started after
2.5 minutes of listening. In addition, although on average there wasn't
a significant difference, the focus level elicited by Apple's playlist
was higher than Silence 60% of the time, starting at 12.5 minutes (<strong>Fig. 7C</strong>), and the focus level elicited by Spotify's playlist was higher than
Silence 27% of the time, starting at 17 minutes (<strong>Fig. 7B</strong>). Focus elicited by Endel's soundscape was higher than Spotify's
playlist in 37% of the time, starting at 6 minutes (<strong>Fig. 7D</strong>).




<Figure src="/deep-dives/pics/deep-dives/modeling_focus_figure7.webp" alt="Comparing brain decoded focus dynamics during the 30 minutes of the Preferred Task." caption={"<strong>Figure 7. Comparing brain decoded focus dynamics during the 30 minutes of the Preferred Task.</strong><br />Each subfigure shows a comparison between two sound streams, while the gray areas are the timings with a significant difference (p&lt;0.05 corrected, see statistical methods for details)."} />




<Figure src="/deep-dives/pics/deep-dives/modeling_focus_table2.webp" alt="Summary of focus time dynamics comparison, showing for each pair the percentage of time and time segments with significant difference." caption={"<strong>Table 2. </strong>Summary of focus time dynamics comparison, showing for each pair the percentage of time and time segments with significant difference (where 100% = 30 minutes)."} />


<h3>Focus levels in response to sounds can be predicted by the sound's properties</h3>


Seeing that background sound had an effect on focus levels, we go
further and ask whether music and soundscapes can be composed according
to a formula to increase focus levels. Meaning, can we understand which
sound properties drive focus well enough to predict focus levels from
only an analysis of the sound properties themselves?




Leveraging the high temporal resolution of the noninvasive brain
measurements, we generated a prediction model which predicts the
brain-based focus level from sound features extracted from the audio
signal. Raw audio files containing the Apple and Spotify sessions were
used to extract different sound properties with a running sliding window
of 30 seconds. The personalized soundscape session (Endel) was not used
in this analysis since the real time streaming did not allow saving the
raw audio files that were consistent across participants. Each sound
property was treated as a unique feature and checked for its
contributory power to the measured average focus level. Supp. Fig. 2
shows the resulting correlations between each sound feature and the
brain based focus level. In total, only 20 features of the 136 features
evaluated were found to have significant correlations (p&lt;0.05, see
Statistical methods).




We next combined multiple sound features to generate a sound data based
model that predicts focus levels (see Methods). <strong>Figure 8</strong> shows the dynamics of the focus predicted from the audio signal alone which
included only properties of the sounds, together with the brain decoded focus
that was derived from high resolution electrical brain measurements (Corr=0.7,
p&lt;1e-4). <strong>Figure 8D</strong> shows that if we threshold our dynamics
to output a binary prediction (low/high focus), the audio model reaches 88%
accuracy in predicting the brain based focus (AUC=0.93).




<Figure src="/deep-dives/pics/deep-dives/modeling_focus_figure8.webp" alt="Results of predicting brain decoded focus from audio features." caption={"<strong>Figure 8. Results of predicting brain decoded focus from audio features. </strong><br /><strong>(A+B)</strong> Dynamics of brain decoded focus (dark blue) and audio decoded focus (light blue), during 30 minutes of the Preferred Task for Apple (A) and Spotify (B). <strong>(C)</strong> Brain decoded focus (y-axis) vs. Audio decoded focus (x-axis) for both playlists (Apple + Spotify). <strong>(D)</strong> Confusion matrix after thresholding the focus predictions to classify between low and high focus. Classification accuracy obtained: 88% (Area under ROC curve: 0.93)."} />




Beyond composing soundscapes for focus, we can also use these prediction
models to rate the focus level of a song and assemble successful
playlists based on existing songs. To demonstrate this, we compared the
song average of the audio decoded output to the brain decoded output. As
shown in <strong>Figure 9B</strong>, there is a correlation of 0.74
between the focus models at the song level (df=16, p=0.0004). <strong>Figure 9A</strong> shows these averages sorted by the brain-based model.




<Figure src="/deep-dives/pics/deep-dives/modeling_focus_figure9.webp" alt="Averaging focus scores for each song." caption={"<strong>Figure 9. Averaging focus scores for each song.</strong><br /><strong>(A)</strong> Sorted focus scores per song obtained by the brain model (brain decoded - blue), next to the focus obtained by the audio model (audio decoded- light blue). <br />&zwj;<strong>(B)</strong> Focus scores per song - brain decoded (y axis) vs. audio decoded (x axis). Pearson correlation between them: Corr(16)=0.74, p=0.0004."} />


<h3>Analysis of a sound's properties can be used to predict its effect on focus</h3>


To gain additional insights about the effects of different sound types
on human focus, we used the trained audio model, and projected songs and
sounds which were not played during the brain recording experiment.
Meaning, we obtained their focus score and dynamics based solely on the
properties of the sounds they contained. We selected sounds that
challenged the validity of the audio model based on their categorical
exclusion from the brain recording experiment. A future approach can
include these different genres as controls for further brain measurement
validation studies. For example, Endel's soundscapes which are not
personalized (taken from the playlist: "<a href="https://open.spotify.com/album/473AYNWq9Lxzs26qjnlVDy?si=iRcJWxMSTvGgba8Hd8TQuw&utm_source=copy-link&dl_branch=1&nd=1">Focus: Calm Clear Morning</a>"), natural sounds which are commonly used for increasing focus (such
as white noise, waves, rain, taken from: <a href="https://mc2method.org/white-noise/">https://mc2method.org/white-noise/</a>), and popular songs from other music genres (classical music,
electronic, pop, rock, jazz and hip hop) were used.




<strong>Figure 10A</strong> shows the predicted focus score based on the
audio model which took into account only the properties of the sounds themselves.
Songs are sorted from the highest focus evoking song (Endel - Three No Paradoxes)
to the lowest (Dr. Dre - What's The Difference). The top two songs are Endel
soundscapes which are not personalized, a finding which strengthens our main
result since it implies that the high focus scores elicited by Endel's soundscape
was not a byproduct of personalization.
<strong>Figure 10B</strong> shows the sorted focus scores averaged across
genres, where notably sounds from classical music and natural sounds contained
properties that predicted high focus levels. In contrast, pop and hip-hop
songs predicted relatively low focus scores. Although we do not have ground
truth focus labels for these songs based on real human brain data, given
the relatively high scores of the sounds which are known to have generated
increased focus in the experimental data, we can conclude that there is a
consistent validity to the model. Future research will gather ground truth
labels for these songs and evaluate the model mathematically in this context.




<Figure src="/deep-dives/pics/deep-dives/modeling_focus_figure10.webp" alt="Projecting new songs into the trained audio model." caption={"<strong>Figure 10. Projecting new songs into the trained audio model.</strong><br /><strong>(A)</strong> Sorted focus scores per song obtained by the audio model, colored by genre. <strong>(B)</strong> Average focus score per genre, sorted from the genre with the lowest score (Hip-hop) to the highest (Classical). <strong>(C)</strong> Average focus variance per genre, sorted from the genre with the lowest variance (Natural) to highest (Electronic). <strong>(D-E)</strong> Focus dynamics for the song with the lowest focus score (D) and the highest (E)."} />




Analyzing the average within-song variance across different genres
reveals that the model predicts the largest variance on average for
electronic sounds (<strong>Fig. 10C</strong>), while the lowest variance
was found for natural sounds. The variance can be interpreted as a range
of focus dynamics, and indeed the focus dynamics of the electronic
sounds might change dramatically during a song (supp. Fig3), confirming
the need for a tool which outputs dynamics with a high temporal
resolution when studying such sound content. <strong>Figure 10D-E</strong> shows the focus dynamics for the song with the lowest focus evoking score
and the highest. The dynamics for all songs can be seen in supp. Fig.3.


<h2>Discussion</h2>


"The soundscape of the world is changing. Modern man is beginning to
inhabit a world with an acoustical environment radically different from
any he has hitherto known" said the composer R. Murray Schafer,
presaging the time we live in now when the sounds available to us
continue to multiply by the day.




As we have an increasing number of options to modulate our auditory
lives by, a handful of take-aways from this study standout:


<h3>Brain-based measurement of focus is possible "in-the-wild"</h3>


Although the effects of sound and music on the human brain can be subtle
in measured brain signals when judging by the changes produced in raw
electromagnetic currents, they are robust and highly quantifiable with
effectively-trained algorithms, as shown here. Classifying emotional and
attentional responses is particularly useful when done at the sub-second
temporal resolution which allows one to track dynamics continuously over
time at the same timescale as the brain functions that impact perception
and behavior. In this study we demonstrated that noninvasive brain
decoding technology is able to deliver this needed resolution with a
high degree of accuracy (approximately 80% match to self-report, <strong>Fig 5C</strong>.). Since there are inherent biases in the subjective self-report for
experience (<strong>Kahneman et al., 1999</strong>; <strong>Mauss &amp; Robinson, 2009</strong>), when mapping physiological signals to self-reported experiences, as
done here, there is an upper boundary for accuracy beyond which the
model will over fit to the self-reported values and incorrectly
represent the information observed in physiological signals. According
to a recent review (<strong>Larradet et al., 2020</strong>) which
summarizes multiple peer-reviewed studies that predict self-reported
emotions from physiological signals, the average accuracy reported was
~82%. Given this average and the experimental conditions here (small
amount of sensors, at home recordings, simple self-report scales), the
achieved accuracy was satisfactory for drawing deeper conclusions on
sound properties, and aligned with state-of-the-art emotion recognition
accuracies in the context of sound as a stimulus (<strong>Tripathi et al., 2017</strong>, 81.41% and 73.35% for 2 classes of Valence and Arousal respectively).




A key benefit of the current approach is that this method of high
temporal resolution brain measurement can be performed reliably outside
of traditional laboratories. In this current study not a single
laboratory or facility was used for data acquisition. Instead, 18-65
year olds across the U.S. received a technology kit in the mail and
experienced music playlists and personalized soundscapes while they
recorded their own brain signals from the comfort of their own homes at
times of day of their choosing. In other words, in their natural
habitat, at their own pace, which lends the research a rare ecological
validity.


<h3>Focus is increased most by personalized, engineered soundscapes</h3>


Within the at-home environment of this study, personalized, engineered
soundscapes were found to be the best at increasing participant's focus
levels (<strong>Fig. 6A</strong>). After 2.5 minutes, on average,
listeners of the personalized soundscapes evaluated experienced a
meaningful increase in their focus level, while for music playlists it
took approximately 15 minutes to gain a similarly appreciable increase (<strong>Fig. 7</strong>). The audio effect on focus levels was found to be task dependent,
where soundscapes increased focus levels in participants who were
working (<strong>Fig 6B</strong>). For participants who were not
working, no significant difference was found. This result suggests that
willful orientation of attention towards work tasks may have created a
brain context especially suited to modification by sound. While engaged
in work, participants may also have been more prone to distraction and
thus more impacted by the positive uplift of sound compared to reading
or playing a game which may have contained more intrinsic motivation to
stay focused on.




One limitation of this current study is that it did not allow us to
disentangle the effects of personalization of sounds on the listener,
since pre-recorded soundscapes were not tested. Equivalently, a
comparison of personalized soundscapes to personalized music playlists,
where audiences either made their own playlist for focus or were allowed
to skip songs whenever they wanted, will likely contribute to a fuller
understanding of how sound properties correlate with emotion and
attention changes. Follow-up research will incorporate these variables.
An additional limitation was the inability to reach conclusions
regarding gender-dependent effects which was at least partially due to
this study's imbalanced data set. Despite efforts to recruit a balanced
group of participants, enrollment was done on a rolling basis and in the
end the female subgroup was statistically underpowered. In future
research, especially for closed loop, real-time testing, balanced
participant sets will be important for reaching more detailed
conclusions.


<h3>Sound preferences and focus effects vary between people</h3>


It is important to emphasize that the results reported here are sound
effects on the average focus levels across a population, and that there
was a large variance in this effect between participants. Evidence for
this large variance can be seen in Supp Fig. 3 and in the age dependency
effect (<strong>Fig 6D-E</strong>), where for the younger audience, all
sounds increased focus while for the older audience, the sounds did not
have any effect. These results are consistent with other studies showing
personal preferences are critical for the improvements possible by
sounds (<strong>Cassidy &amp; Macdonald, 2009</strong>; <strong>Huang &amp; Shih, 2011</strong>; <strong>Mori et al., 2014</strong>). Due to this variety observed
together with the highest focus being elicited by the personalized
soundscapes, a next step will include closed-loop selections of sounds,
where iterative sound testing is used per person to identify the
significant parameters for maximizing focus for that person.




Personalized soundscapes specifically, and personalized audio in
general, should be investigated further for their capacity to increase
productivity, creativity and well-being as these attributes of human
experience are associated with one's ability to focus. For clinical
populations as well, for example children with ADHD, the tailoring of
sounds for this purpose of increased focus can be particularly
impactful. It is possible that the seamlessness of the personalized
soundscapes tested here, which played continuously without gaps in the
sound like the music playlists had between songs, was a critical part of
the observed effect on focus. At every juncture of the experience there
is more to be learned, but at a high level, a main lesson of this study
is that there is a strong need for personalization of sound in order to
most effectively achieve functional goals like increasing focus.


<h3>Brain decoded focus data enabled a new predictive model based on sound data alone</h3>


Leveraging the high temporal resolution of the brain decoded dynamics, a
focus prediction model based on the physical properties of sounds was
successfully trained, resulting in an accuracy score of 88% in
predicting the brain decoded focus score from an audio decomposition
that assessed 136 different properties of sounds as unique features (<strong>Fig. 8</strong>). This model enabled a further examination of how sounds and different
genres effects focus and allowed testing additional conditions, such as
pre-recorded soundscapes and commonly used background sounds (e.g. white
noise), as well as other genres (pop, rock, jazz, etc). We found that
the model predicted the highest focus score to classical music, followed
by engineered soundscapes and natural sounds. These results complement
previous studies which showed natural sounds and classical music are
beneficial for learning and concentration (<strong>Angwin et al., 2017</strong>; <strong>Davies, 2000</strong>; <strong>DeLoach et al., 2015</strong>;
H. <strong>Liu et al., 2021</strong>).




In contrast, the models predicted that genres such as pop and hip-hop
produce lower focus scores (<strong>Fig 10A-B</strong>). It is possible
that these sounds contain more distractors that attract attention away
from other objects of attention, or that they contain types of sounds
that the brain requires more resources to process (depending on familiar
patterns, surprises and more), leading to less resources available to
perform other tasks. Sounds in these genres may also activate the reward
system more (<strong>Gold et al., 2019</strong>; <strong>Salimpoor et al., 2015</strong>), which can increase motivation and improve learning of the songs
themselves rather than orientation towards other tasks. These sounds may
be optimal for driving focus on the songs in other words, rather than
focus on other things. Understanding the brain mechanisms that underlie
the modified focus is beyond the scope of this work, but the mapping
found here can potentially provide fruitful directions for future brain
imaging experiments that are equipped to answer these questions.




The analysis here demonstrates a process in which we utilize the
temporal resolution of the brain sensing technology to generate a
product where the neurotechnology is eventually out of the loop,
resulting in a stand alone sound model which gets as an input a raw
audio file and outputs a predicted focus score. This model can be used
independently to generate focus playlists or to compose optimal
soundscapes, and can further be improved by expanding to populations
outside the U.S. and different age groups.




Pythagoras, who first identified the mathematical connection between a
string's length and it's pitch, believed that the whole cosmos was a
form of musical composition (<strong>James, 1995</strong>). We too see
the rich mathematical models obtained in this study, by mapping sound
properties to human experience, as a glimpse into the natural laws
governing how we feel and think. The better these laws can be
understood, the more empowered individuals will be to modulate their
sound environments to suit their goals and states of mind. There remains
much to figure out. While we as a species continue to cause a "shift in
the sensorium," we simultaneously experience that shift all over daily
life and it is not clear where we as a species are headed. This study
showed that sounds have a distinct effect on our focus, and paves the
way for designing sounds to help us focus better in the future.


<h2>Conclusions</h2>


We studied the effects of sound on human focus levels using noninvasive
brain decoding technology and to gain a better understanding of the
optimal sound properties for increasing focus levels in listeners. We
combined a custom app, portable EEG-measuring headbands, and brain
decoding technology that enabled us to obtain high temporal resolution
focus dynamics from participants at home. Using the brain decoded focus
dynamics, we then analyzed how various properties of sound affected
focus levels in different tasks.



We found that while performing a self-paced task for a long period of
time (such as working), personalized soundscapes increased focus the
most relative to silence. Curated playlists of pre-recorded songs by
Apple and Spotify also increased focus during specific time intervals,
especially for the youngest audience demographic. Large variance in
response profiles across participants, together with task and age
dependent effects, suggest that personalizing sounds in real-time may be
the best strategy overall for producing focus in a given listener.



Finally, we generated a sound property based focus model which
successfully predicts the brain decoded focus scores using only an audio
file as input. Using this model, we extracted predicted focus scores
from new songs based on audio decomposition and performed a genre
analysis to develop new intuitions about the findings and the source of
focus-producing sound content. We found that based on our model,
engineered soundscapes and classical music are the best for increasing
focus, while pop and hip-hop music are the worst.



The approach taken here can be adapted to include other emotions (e.g.
enjoyment, stress, happiness, etc.), attentional parameters ('Flow
state,' memory formation, etc.) and can be used to assess additional
content as well (e.g. visual, ambient, olfactory, etc.), including
interactive gaming and e-learning where personalization and high
temporal resolution experience measures may be especially beneficial.


<h2>Data Availability Statement</h2>


The dataset for this study is available through an open Git repository (<a href="https://github.com/arctop/Measuring-And-Modeling-The-Effect-of-Audio-On-Human-Focus-In-Everyday-Environments-Using-BCI">link</a>). Data includes the brain decoded focus dynamics for each participant
together with scripts that run the statistical tests.


<h2>Author Contributions</h2>


A.H., R.K, N.E, E.K, and D.F designed the experiment, A.H analyzed the
data, R.K, N.E, and D.F. advised on data analysis and statistics. S.K
and E.K developed the app and software platform for data collection. A.H
and D.F wrote the paper, R.K and E.K revised the paper. All authors
approved the work for publication.


<h2>Conflict of Interest</h2>


This study received funding from Arctop Inc. and Endel Sound GmbH. The
funders had the following involvement with the study: Arctop Inc. was
involved in study design, collection, analysis, interpretation of data,
the writing of this article and the decision to submit it for
publication. Endel Sound GmbH was involved in study design and provided
audio stimuli used in the experiment. All authors declare no other
competing interests.


<h2>Acknowledgments</h2>


We would like to thank Hillel Pratt, Kevin Liu, Alexander Kopanev,
Warner Music, Sony Music, Endel, and Universal Music for providing audio
content, data, and support in conducting this study and advancing
theoretical and applied aspects of the research.


<h2>Funding</h2>


This study received funding from Arctop Inc. and Endel Sound GmbH. Endel
Sound GmbH was not involved in data collection, analysis, interpretation
of data, the writing of this article or the decision to submit it for
publication. All authors declare no other competing interests.




<References items={[{"html":"Abiri, R., Borhani, S., Jiang, Y., &amp; Zhao, X. (2019). Decoding Attentional State to Faces and Scenes Using EEG Brainwaves. <em>Complexity</em>, 2019, e6862031. <a href=\"https://doi.org/10.1155/2019/6862031\">https://doi.org/10.1155/2019/6862031</a>"},{"html":"Angwin, A. J., Wilson, W. J., Arnott, W. L., Signorini, A., Barry, R. J., &amp; Copland, D. A. (2017). White noise enhances new-word learning in healthy adults. <em>Scientific Reports</em>, 7(1), 13045. <a href=\"https://doi.org/10.1038/s41598-017-13383-3\">https://doi.org/10.1038/s41598-017-13383-3</a>"},{"html":"Asif, A., Majid, M., &amp; Anwar, S. M. (2019). Human stress classification using EEG signals in response to music tracks. <em>Computers in Biology and Medicine</em>, 107, 182–196."},{"html":"Bhatti, A. M., Majid, M., Anwar, S. M., &amp; Khan, B. (2016). Human emotion recognition and analysis in response to audio music using brain signals. <em>Computers in Human Behavior</em>, 65, 267–275. <a href=\"https://doi.org/10.1016/j.chb.2016.08.029\">https://doi.org/10.1016/j.chb.2016.08.029</a>"},{"html":"Bird, J. J., Ekart, A., Buckingham, C. D., &amp; Faria, D. R. (2019). Mental emotional sentiment classification with an eeg-based brain-machine interface. <em>Proceedings of TheInternational Conference on Digital Image and Signal Processing (DISP'19).</em>"},{"html":"Broday-Dvir, R., Grossman, S., Furman-Haran, E., &amp; Malach, R. (2018). Quenching of spontaneous fluctuations by attention in human visual cortex. <em>NeuroImage</em>, 171, 84–98. <a href=\"https://doi.org/10.1016/j.neuroimage.2017.12.089\">https://doi.org/10.1016/j.neuroimage.2017.12.089</a>"},{"html":"Brotzer, J. M., Mosqueda, E. R., &amp; Gorro, K. (2019). Predicting emotion in music through audio pattern analysis. <em>IOP Conference Series: Materials Science and Engineering</em>, 482, 012021. <a href=\"https://doi.org/10.1088/1757-899X/482/1/012021\">https://doi.org/10.1088/1757-899X/482/1/012021</a>"},{"html":"Cassidy, G., &amp; Macdonald, R. (2009). The effects of music choice on task performance: A study of the impact of self-selected and experimenter-selected music on driving game performance and experience. <em>Musicae Scientiae</em>, 13(2), 357–386. <a href=\"https://doi.org/10.1177/102986490901300207\">https://doi.org/10.1177/102986490901300207</a>"},{"html":"Chanda, M. L., &amp; Levitin, D. J. (2013). The neurochemistry of music. <em>Trends in Cognitive Sciences</em>, 17(4), 179–193. <a href=\"https://doi.org/10.1016/j.tics.2013.02.007\">https://doi.org/10.1016/j.tics.2013.02.007</a>"},{"html":"Cheung, V. K. M., Harrison, P. M. C., Meyer, L., Pearce, M. T., Haynes, J.-D., &amp; Koelsch, S. (2019). Uncertainty and Surprise Jointly Predict Musical Pleasure and Amygdala, Hippocampus, and Auditory Cortex Activity. <em>Current Biology</em>, 29(23), 4084–4092.e4. <a href=\"https://doi.org/10.1016/j.cub.2019.09.067\">https://doi.org/10.1016/j.cub.2019.09.067</a>"},{"html":"Chou, P. T.-M. (2010). Attention Drainage Effect: How Background Music Effects Concentration in Taiwanese College Students. <em>Journal of the Scholarship of Teaching and Learning</em>, 10(1), 36–46."},{"html":"Cowen, A. S., &amp; Keltner, D. (2017). Self-report captures 27 distinct categories of emotion bridged by continuous gradients. <em>Proceedings of the National Academy of Sciences</em>, 114(38), E7900–E7909."},{"html":"Cunningham, S., Ridley, H., Weinel, J., &amp; Picking, R. (2020). Supervised machine learning for audio emotion recognition. <em>Personal and Ubiquitous Computing</em>, 1–14."},{"html":"Davies, M. A. (2000). Learning … the Beat Goes on. <em>Childhood Education</em>, 76(3), 148–153. <a href=\"https://doi.org/10.1080/00094056.2000.10522096\">https://doi.org/10.1080/00094056.2000.10522096</a>"},{"html":"Davis, W. B., &amp; Thaut, M. H. (1989). The Influence of Preferred Relaxing Music on Measures of State Anxiety, Relaxation, and Physiological Responses 1. <em>Journal of Music Therapy</em>, 26(4), 168–187. <a href=\"https://doi.org/10.1093/jmt/26.4.168\">https://doi.org/10.1093/jmt/26.4.168</a>"},{"html":"de la Mora Velasco, E., &amp; Hirumi, A. (2020). The effects of background music on learning: A systematic review of literature to guide future research and practice. <em>Educational Technology Research and Development</em>, 68(6), 2817–2837. <a href=\"https://doi.org/10.1007/s11423-020-09783-4\">https://doi.org/10.1007/s11423-020-09783-4</a>"},{"html":"DeLoach, A. G., Carter, J. P., &amp; Braasch, J. (2015). Tuning the cognitive environment: Sound masking with “natural” sounds in open-plan offices. <em>The Journal of the Acoustical Society of America</em>, 137(4), 2291–2291. <a href=\"https://doi.org/10.1121/1.4920363\">https://doi.org/10.1121/1.4920363</a>"},{"html":"Faller, J., Cummings, J., Saproo, S., &amp; Sajda, P. (2019). Regulation of arousal via online neurofeedback improves human performance in a demanding sensory-motor task. <em>Proceedings of the National Academy of Sciences</em>, 116(13), 6482–6490."},{"html":"Gao, C., Fillmore, P., &amp; Scullin, M. K. (2020). Classical music, educational learning, and slow wave sleep: A targeted memory reactivation experiment. <em>Neurobiology of Learning and Memory</em>, 171, 107206. <a href=\"https://doi.org/10.1016/j.nlm.2020.107206\">https://doi.org/10.1016/j.nlm.2020.107206</a>"},{"html":"Giannakopoulos, T. (2015). pyAudioAnalysis: An Open-Source Python Library for Audio Signal Analysis. <em>PLOS ONE</em>, 10(12), e0144610. <a href=\"https://doi.org/10.1371/journal.pone.0144610\">https://doi.org/10.1371/journal.pone.0144610</a>"},{"html":"Gold, B. P., Mas-Herrero, E., Zeighami, Y., Benovoy, M., Dagher, A., &amp; Zatorre, R. J. (2019). Musical reward prediction errors engage the nucleus accumbens and motivate learning. <em>Proceedings of the National Academy of Sciences</em>, 116(8), 3310–3315. <a href=\"https://doi.org/10.1073/pnas.1809855116\">https://doi.org/10.1073/pnas.1809855116</a>"},{"html":"D. D. Schmorrow &amp; C. M. Fidopiastis González, V. M., Robbes, R., Góngora, G., &amp; Medina, S. (2015). Measuring Concentration While Programming with Low-Cost BCI Devices: Differences Between Debugging and Creativity Tasks. In D. D. Schmorrow &amp; C. M. Fidopiastis (Eds.), <em>Foundations of Augmented Cognition</em> (pp. 605–615). Springer International Publishing. <a href=\"https://doi.org/10.1007/978-3-319-20816-9_58\">https://doi.org/10.1007/978-3-319-20816-9_58</a>&zwj;"},{"html":"Hallam, S., Price, J., &amp; Katsarou, G. (2002). The Effects of Background Music on Primary School Pupils’ Task Performance. <em>Educational Studies</em>, 28(2), 111–122. <a href=\"https://doi.org/10.1080/03055690220124551\">https://doi.org/10.1080/03055690220124551</a>&zwj;"},{"html":"Hamadicharef, B., Zhang, H., Guan, C., Wang, C., Phua, K. S., Tee, K. P., &amp; Ang, K. K. (2009). Learning EEG-based spectral-spatial patterns for attention level measurement. <em>2009 IEEE International Symposium on Circuits and Systems</em>, 1465–1468."},{"html":"Hizlisoy, S., Yildirim, S., &amp; Tufekci, Z. (2021). Music emotion recognition using convolutional long short term memory deep neural networks. <em>Engineering Science and Technology, an International Journal</em>, 24(3), 760–767. <a href=\"https://doi.org/10.1016/j.jestch.2020.10.009\">https://doi.org/10.1016/j.jestch.2020.10.009</a>&zwj;"},{"html":"Hu, J. (2017). Automated Detection of Driver Fatigue Based on AdaBoost Classifier with EEG Signals. <em>Frontiers in Computational Neuroscience</em>, 0. <a href=\"https://doi.org/10.3389/fncom.2017.00072\">https://doi.org/10.3389/fncom.2017.00072</a>&zwj;"},{"html":"Huang, R.-H., &amp; Shih, Y.-N. (2011). Effects of background music on concentration of workers. <em>Work</em>, 38(4), 383–387. <a href=\"https://doi.org/10.3233/WOR-2011-1141\">https://doi.org/10.3233/WOR-2011-1141</a>&zwj;"},{"html":"Huron, D. B. (2006). <em>Sweet Anticipation: Music and the Psychology of Expectation</em>. MIT Press."},{"html":"James, J. (1995). The Music of the Spheres: Music, Science, and the Natural Order of the Universe. <em>Copernicus.</em> <a href=\"https://www.springer.com/gp/book/9780387944746\">https://www.springer.com/gp/book/9780387944746</a>&zwj;"},{"html":"Jung, T.-P., Makeig, S., Stensmo, M., &amp; Sejnowski, T. J. (1997). Estimating alertness from the EEG power spectrum. <em>IEEE Transactions on Biomedical Engineering</em>, 44(1), 60–69."},{"html":"Kahneman, D., Diener, E., &amp; Schwarz, N. (1999). <em>Well-being: Foundations of hedonic psychology.</em> Russell Sage Foundation."},{"html":"Kumar, S., von Kriegstein, K., Friston, K., &amp; Griffiths, T. D. (2012). Features versus feelings: Dissociable representations of the acoustic features and valence of aversive sounds. <em>Journal of Neuroscience</em>, 32(41), 14184–14192."},{"html":"Larradet, F., Niewiadomski, R., Barresi, G., Caldwell, D. G., &amp; Mattos, L. S. (2020). Toward Emotion Recognition From Physiological Signals in the Wild: Approaching the Methodological Issues in Real-Life Data Collection. <em>Frontiers in Psychology</em>, 11, 1111. <a href=\"https://doi.org/10.3389/fpsyg.2020.01111\">https://doi.org/10.3389/fpsyg.2020.01111</a>&zwj;"},{"html":"Larsen, R. J., &amp; Diener, E. (1992). <em>Promises and problems with the circumplex model of emotion.</em>&zwj;"},{"html":"Levitin, D. J. (2006). <em>This is your brain on music: The science of a human obsession</em>. Penguin."},{"html":"Levitin, D. J., Chordia, P., &amp; Menon, V. (2012). Musical rhythm spectra from Bach to Joplin obey a 1/f power law. <em>Proceedings of the National Academy of Sciences</em>, 109(10), 3716–3720. <a href=\"https://doi.org/10.1073/pnas.1113828109\">https://doi.org/10.1073/pnas.1113828109</a>&zwj;"},{"html":"Lin, Y.-P., Jao, P.-K., &amp; Yang, Y.-H. (2017). Improving Cross-Day EEG-Based Emotion Classification Using Robust Principal Component Analysis. <em>Frontiers in Computational Neuroscience</em>, 0. <a href=\"https://doi.org/10.3389/fncom.2017.00064\">https://doi.org/10.3389/fncom.2017.00064</a>&zwj;"},{"html":"Liu, H., He, H., &amp; Qin, J. (2021). Does background sounds distort concentration and verbal reasoning performance in open-plan office? <em>Applied Acoustics</em>, 172, 107577. <a href=\"https://doi.org/10.1016/j.apacoust.2020.107577\">https://doi.org/10.1016/j.apacoust.2020.107577</a>&zwj;"},{"html":"Liu, N.-H., Chiang, C.-Y., &amp; Chu, H.-C. (2013). Recognizing the Degree of Human Attention Using EEG Signals from Mobile Sensors. <em>Sensors (Basel, Switzerland)</em>, 13(8), 10273–10286. <a href=\"https://doi.org/10.3390/s130810273\">https://doi.org/10.3390/s130810273</a>&zwj;"},{"html":"Mauss, I. B., &amp; Robinson, M. D. (2009). Measures of emotion: A review. <em>Cognition and Emotion</em>, 23(2), 209–237. <a href=\"https://doi.org/10.1080/02699930802204677\">https://doi.org/10.1080/02699930802204677</a>&zwj;"},{"html":"Micoulaud-Franchi, J.-A., Geoffroy, P. A., Fond, G., Lopez, R., Bioulac, S., &amp; Philip, P. (2014). EEG neurofeedback treatments in children with ADHD: An updated meta-analysis of randomized controlled trials. <em>Frontiers in Human Neuroscience</em>, 0. <a href=\"https://doi.org/10.3389/fnhum.2014.00906\">https://doi.org/10.3389/fnhum.2014.00906</a>&zwj;"},{"html":"Mori, F., Naghsh, F. A., &amp; Tezuka, T. (2014). The Effect of Music on the Level of Mental Concentration and its Temporal Change. <em>Proceedings of the 6th International Conference on Computer Supported Education</em> - Volume 1, 34–42. <a href=\"https://doi.org/10.5220/0004791100340042\">https://doi.org/10.5220/0004791100340042</a>&zwj;"},{"html":"Nia, H. T., Jain, A. D., Liu, Y., Alam, M.-R., Barnas, R., &amp; Makris, N. C. (2015). The evolution of air resonance power efficiency in the violin and its ancestors. <em>Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences</em>, 471(2175), 20140905. <a href=\"https://doi.org/10.1098/rspa.2014.0905\">https://doi.org/10.1098/rspa.2014.0905</a>&zwj;"},{"html":"Perez-Valero, E., Vaquero-Blasco, M. A., Lopez-Gordo, M. A., &amp; Morillas, C. (2021). Quantitative Assessment of Stress Through EEG During a Virtual Reality Stress-Relax Session. <em>Frontiers in Computational Neuroscience</em>, 0. <a href=\"https://doi.org/10.3389/fncom.2021.684423\">https://doi.org/10.3389/fncom.2021.684423</a>&zwj;"},{"html":"J. A. Jacko Rebolledo-Mendez, G., Dunwell, I., Martínez-Mirón, E. A., Vargas-Cerdán, M. D., de Freitas, S., Liarokapis, F., &amp; García-Gaona, A. R. (2009). Assessing NeuroSky’s Usability to Detect Attention Levels in an Assessment Exercise. In J. A. Jacko (Ed.), <em>Human-Computer Interaction. New Trends</em> (pp. 149–158). Springer. <a href=\"https://doi.org/10.1007/978-3-642-02574-7_17\">https://doi.org/10.1007/978-3-642-02574-7_17</a>&zwj;"},{"html":"Sacks, O. (2010). <em>Musicophilia: Tales of music and the brain</em>. Vintage Canada."},{"html":"Salimpoor, V. N., Zald, D. H., Zatorre, R. J., Dagher, A., &amp; McIntosh, A. R. (2015). Predictions and the brain: How musical sounds become rewarding. <em>Trends in Cognitive Sciences</em>, 19(2), 86–91. <a href=\"https://doi.org/10.1016/j.tics.2014.12.001\">https://doi.org/10.1016/j.tics.2014.12.001</a>&zwj;"},{"html":"Schreiber, C. A., &amp; Kahneman, D. (2000). Determinants of the remembered utility of aversive sounds. <em>Journal of Experimental Psychology: General</em>, 129(1), 27."},{"html":"Shahabi, H., &amp; Moghimi, S. (2016). Toward automatic detection of brain responses to emotional music through analysis of EEG effective connectivity. <em>Computers in Human Behavior</em>, 58, 231–239. <a href=\"https://doi.org/10.1016/j.chb.2016.01.005\">https://doi.org/10.1016/j.chb.2016.01.005</a>&zwj;"},{"html":"Shih, Y.-N., Huang, R.-H., &amp; Chiang, H.-Y. (2012). Background music: Effects on attention performance. <em>Work</em>, 42(4), 573–578. <a href=\"https://doi.org/10.3233/WOR-2012-1410\">https://doi.org/10.3233/WOR-2012-1410</a>&zwj;"},{"html":"Tripathi, S., Acharya, S., Sharma, R. D., Mittal, S., &amp; Bhattacharya, S. (2017). Using Deep and Convolutional Neural Networks for Accurate Emotion Classification on DEAP Dataset. <em>Twenty-Ninth IAAI Conference</em>."},{"html":"Tuckute, G., Hansen, S. T., Kjaer, T. W., &amp; Hansen, L. K. (2021). Real-Time Decoding of Attentional States Using Closed-Loop EEG Neurofeedback. <em>Neural Computation</em>, 33(4), 967–1004. <a href=\"https://doi.org/10.1162/neco_a_01363\">https://doi.org/10.1162/neco_a_01363</a>&zwj;"},{"html":"Vempala, N. N., &amp; Russo, F. A. (2012). Predicting emotion from music audio features using neural networks. <em>Proceedings of the 9th International Symposium on Computer Music Modeling and Retrieval (CMMR)</em>, 336–343."},{"html":"Washburne, C. (2020). “More Cowbell”: Latin Jazz in the Twenty-First Century. In <em>Latin Jazz</em>. Oxford University Press. <a href=\"https://doi.org/10.1093/oso/9780195371628.003.0007\">https://doi.org/10.1093/oso/9780195371628.003.0007</a>&zwj;"},{"html":"Yang, Y.-H., Lin, Y.-C., Su, Y.-F., &amp; Chen, H. H. (2008). A Regression Approach to Music Emotion Recognition. <em>IEEE Transactions on Audio, Speech, and Language Processing</em>, 16(2), 448–457. <a href=\"https://doi.org/10.1109/TASL.2007.911513\">https://doi.org/10.1109/TASL.2007.911513</a>&zwj;"},{"html":"Zald, D. H., &amp; Pardo, J. V. (2002). The neural correlates of aversive auditory stimulation. <em>Neuroimage</em>, 16(3), 746–753."}]} />