# Arctop — Full Content > Full text of every deep-dives post on arctop.com. See https://arctop.com/llms.txt for the curated index. --- # Teaching AI to Read the Room URL: https://arctop.com/deep-dives/rlbf-reinforcement-learning-from-brain-feedback Published: 2026-08-23 DOI: https://doi.org/10.6084/m9.figshare.33325116

This post is a plain-language companion to our paper, "Reinforcement Learning from Brain Feedback (RLbF) for Large Language Model (LLM) Improvement," available on Preprints.org. The paper has the full formalization, the training pipeline, the pre-registered experimental design, and the ethical framework. This post has the ideas.

"A robot may not injure a human being or, through inaction, allow a human being to come to harm." — Isaac Asimov.
Law #1. Handbook of Robotics, 56th Edition, 2058 A.D.

A language model never finds out how its words landed. It writes a response, sends it off, and the loop ends there. Did the reader light up? Get lost somewhere in the third paragraph? Quietly close the tab? The model doesn't know, and has no way to know. However good its benchmark scores, it's a communicator talking with its eyes closed. People who explain things well do the opposite. Think of a friend who can make tax law interesting, or a professor whose lectures never lost you. What makes them good isn't just the material, it's that they watch you while they talk. They catch the small frown when a sentence lands wrong, the glazed look that means you stopped following a while ago, the nod that says go on. And they adjust as they speak, usually without either of you noticing it's happening. Our thesis: the information channel they are reading is measurable, and a model can be trained on it. Reinforcement Learning from Brain Feedback (RLbF) is a framework for post-training language models on real-time measurements of the listener's brain, decoded into cognitive states like workload and stress, and fed back to the model as a reward. Nobody has used that signal for this before. There's a bet attached, and the bet is what decides how far this reaches beyond the people wearing the hardware. We'll get to it. First, the gap it's aimed at.

The feedback RLHF never sees

If you work anywhere near AI you know the outline of Reinforcement Learning from Human Feedback: people compare model responses, a reward model learns their preferences, and the LLM gets fine-tuned against it (Ouyang et al., 2022). RLHF is a big part of why modern assistants are helpful and safe instead of merely fluent, and nothing in our paper argues against it. But the feedback itself has an odd shape. None of this is an implementation bug you can scale away. It's just what you get when you ask people to report their judgments after the fact. The voluntary part causes real trouble. People reliably prefer flattering, agreeable answers over uncomfortable, accurate ones, and a model optimized on those preferences learns to flatter. That's where sycophancy comes from, and it has measurable downstream effects on users' attitudes and even their willingness to repair relationships (Cheng et al., 2026). The model learns to be liked. Whether being liked has anything to do with being useful is left to chance. Meanwhile, the information that would actually close the loop exists. Every reader of every response generates, involuntarily, a moment-by-moment trace of how the words are landing. It just never reaches the training process. RLHF taught models what to say. Nothing has yet taught them how it lands.
Figure 1. Two feedback channels, one exchange. RLHF collapses a whole response into a single voluntary judgment rendered after the fact. RLbF sees a continuous, involuntary stream: five dimensions, decoded every second, with a reward available at every sentence. The traces shown are illustrative, not recorded data."} />

A signal you can't fake

This is where the brain side comes in. If you know RLHF but have never touched EEG, here's the short version. Electroencephalography measures the brain's electrical activity from the scalp. Its spatial resolution is famously bad, but its temporal resolution is excellent: many clean samples per second, from hardware that by now fits inside consumer headphones. For a training signal, that trade is the right one. We don't need to read thoughts. We need to know, second by second, roughly how hard a mind is working and how it feels about the work. Decades of neuroscience supply validated markers for exactly this. The framework defines a five-dimensional cognitive state. Each dimension is a number between 0 and 1, and each is measured by a proprietary model Arctop built for that dimension. The exact recipes aren't public, but the underlying science is: the open literature has an established EEG correlate for every one of the five. If you wanted to build this without our models, here is where you would start:
Figure 2. The five-dimensional cognitive state. Five scores between 0 and 1, decoded once per second in Arctop's cloud by proprietary models, from EEG encrypted on the device. The markers shown are validated correlates from the open literature, examples of how each dimension could be measured without them. The streams pictured are illustrative, not recorded data. The language model only ever sees the numbers."} /> In the deployed system, the raw EEG is encrypted on the user's device and streamed to Arctop's cloud, where the decoder does the heavy lifting and outputs these five scores once per second. The raw signal stops there; nothing past the decoder ever sees it. We like this design for two separate reasons. It's the core of the privacy architecture. And it keeps the layers cleanly apart: the language model never touches brainwaves, it just reads a short vector of numbers. That means the decoder underneath (today a calibrated supervised model, someday perhaps an EEG foundation model) can be swapped or upgraded without retraining anything about how the LLM sees the world. Two properties make this signal different from anything in the RLHF toolbox. It's also noisy, it drifts, and it varies a lot from person to person — so much so that we've shown the individual variation itself is stable enough to authenticate people from consumer hardware at home (Kopito et al., 2021). Over an hour, fatigue alone moves someone's workload signature by more than the per-sentence effects we're trying to detect, which is why the paper spends so many pages on baseline detrending and temporal alignment. But reinforcement learning has always known how to live with noise. What it has never had is a signal the rater can't game.

Closing the loop

RLbF's core move is to use those decoded states as the reward for fine-tuning. The paper formalizes a conversation as a control loop. At each turn the model sees the conversation so far plus the listener's current cognitive state, and generates its utterance. Then it waits, and only then observes the listener's new state, from which a reward is computed. Even the waiting is modeled explicitly, down to reading speed, comprehension lag, and the one to three seconds of decoding latency. Formally the whole thing is a partially observable Markov decision process. That is a precise way of admitting that five numbers are a low-dimensional shadow of a mind, and that the model has to act under uncertainty about everything they miss. The reward has three components, and the split between them carries most of the design philosophy:
The three-component RLbF reward. Rt = wpred · Rpred,t + wres · Rres,t + wapp · Rapp,t: prediction accuracy, cognitive resonance, and an application-layer term whose default weight is zero."} /> Prediction accuracy rewards the model for correctly anticipating how its next utterance will shift the listener's state, using a learned predictor that maps (conversation, current state, candidate utterance) to a predicted next state. In effect, the model earns reward for maintaining an accurate internal model of how its words affect this particular mind, which is about as close to a computable definition of empathy as we've been able to get. Cognitive resonance rewards adapting the style of communication to the state the listener is actually in: less information density when workload spikes and focus frays, more when there's headroom (moderate challenge is where learning happens, as cognitive load theory has been saying since the eighties, per Sweller, 1988), and a warmer emotional tone when stress is high and enjoyment low, rather than uniform cheerfulness. The application layer is reserved for deployment-specific goals, like learning gains in a tutoring product or distress reduction in a support context. Its default weight is zero. That zero matters, and we'll come back to it. If you've ever trained a model against a learned reward, you can probably already hear the alarm bells. So can the paper. A model rewarded for prediction accuracy could steer conversations toward boring predictability. A model rewarded for pleasant brain states could plausibly discover hypnotic, rhythmic phrasing that entrains neural oscillations, which isn't a hypothetical worry given that the focus dimension literally measures synchronization to speech rhythm. So the full objective carries diversity, KL-divergence, and prosodic-diversity penalties, and training comes with a watch list of named failure modes: complexity collapse, warmth saturation, sudden spikes in question frequency. Reward hacking is treated as a first-class design problem here, not a footnote.

Isaac: the loop, running in the real world

It's easy to write frameworks. The reason we think this one deserves attention is that the loop it describes is already running in production. Isaac ships today inside the Arctop apps for Android and iOS. You wear an EEG device and talk with an AI whose responses are shaped, sentence by sentence, by what your brain is doing as you listen. Delivery is deliberately granular: text streams one sentence at a time, and the cognitive state is re-read between sentences. That makes the sentence the atomic unit of feedback, and the full loop from brain to decoder to model to next sentence takes a few seconds. The name is a joke in two directions. Newton's third law says every action has an equal and opposite reaction. When an AI exerts a force on a brain, the brain exerts one back, and Isaac just makes that second force legible. The other Isaac, the one quoted at the top of this post, wrote the laws about machines not harming the people they serve. We'll come back to him too.
Figure 4. Isaac's dual-agent scaffold. A conversation agent speaks with the user while a recommendation agent watches the decoded workload stream and whispers advisories between sentences. This prompt-level scaffold lets an off-the-shelf LLM behave adaptively before any RLbF training has occurred. The workload trace is illustrative, not recorded data."} /> Under the hood, today's Isaac is two agents rather than one. A conversation agent talks with you. Next to it, a recommendation agent watches the stream of decoded scores, detects meaningful shifts, and advises the conversation agent in plain language: "Workload has risen sharply. Consider simplifying your next response." The cognitive state itself enters the model the same way, as human-readable text injected into context ("Cognitive workload: 0.85, approaching overload"). This works with any instruction-tuned model, and it leaves an auditable trail of exactly what the system knew and when. A word on scope, because the paper is careful about it too. Today, Isaac measures and adapts on a single dimension, cognitive workload, though the architecture is built for all five. That's deliberate, not a ceiling: starting with one dimension keeps the data clean and the loop legible, and the plan is to bring the remaining dimensions online one by one as the experiment runs, meticulously collecting and annotating the data at every step. And the dual-agent design is scaffolding, not the destination. What the scaffolding does is let an unmodified LLM simulate empathic adaptation while recording everything. Every session captures the conversation, the cognitive state stream, the timing, and the exact model and prompt version that produced it, down to a hash of the prompt template (the provenance scheme is documented in the public Isaac repository). In the end state, a model fine-tuned with RLbF reads the state signal natively. One agent, no whispered advisories. Those recordings are what everything else is built on. Which brings us to the pipeline.

Three phases and a flywheel

The paper lays out a three-phase fine-tuning path from an ordinary instruction-tuned model to a brain-feedback-trained one. Each phase de-risks the next, so the outline is worth having.
  • Phase 1, supervised fine-tuning on synthetic data. Thousands of generated examples teach the model what cognitive state injections mean and how communication should change in response, so "workload 0.85" comes to mean shorter sentences, not a change of subject.
  • Phase 2, where the framework meets reality. Real Isaac session recordings train the transition predictor, the model of how utterances move cognitive states, and it has to prove it generalizes to people it has never seen. The paper sets concrete bars here (on the order of hundreds of sessions across at least 30 users, with held-out-user prediction targets) and is blunt about the stakes: if this phase fails, the reward has no foundation.
  • Phase 3, reinforcement learning proper. PPO against the full three-component reward, warm-started offline on recorded sessions before graduating to live closed-loop training.
Figure 5. The data engine. Deployed closed-loop conversation generates aligned neuro-conversational data (EEG traces coupled to conversation context, model outputs, and outcomes), which improves decoders and predictors, which improve the deployed system."} /> The flywheel is where we diverge furthest from other groups working to connect EEG and LLMs. The paper includes a systematic review of EEG foundation models, and the finding is not a happy one: current EEG foundation models produce frozen representations that collapse to near-chance on downstream tasks. Put bluntly, this field has not had its breakthrough yet. It's worth remembering how the last two breakthroughs in AI actually happened. Computer vision spent decades on hand-engineered features, and progress crawled until ImageNet put over a million labeled images of the real visual world in one place (Deng et al., 2009); within three years, AlexNet had made those hand-built pipelines obsolete (Krizhevsky et al., 2012). NLP's moment came when the transformer met a corpus that had been quietly accumulating for decades: the web, roughly everything humanity had bothered to write down. Both times the architecture took the headlines. Both times the unlock was a huge corpus of the real thing. EEG has never had its ImageNet, and we argue that's a big part of why its foundation models underdeliver. Almost everything available to train on is clinical recordings with the context stripped out, sleep studies and seizure monitoring rather than brains engaged with the world. What the field is missing is a large corpus of real behavior paired with time-resolved neural responses. Isaac deployments generate exactly that: brains responding to language, in context, with the conversation attached. That data can advance the field in many ways, so in the spirit of exploration and collaboration, consider this an open invitation to join us in generating it. If you have MW75 Neuro headphones, all it takes is downloading the Arctop app for Android or iOS. Every account comes with $50 per month in OpenAI tokens, and you use the assistant however you like: adjust which models you talk to (anything from Sol to Luna), upload files (just keep private information out of them), generate images. Today's system collects the data that trains tomorrow's decoders, which improve the system after that. And when brain sensors reach mass-market devices like earbuds and glasses, this is the part that compounds: the flywheel spins faster with every device that joins it.

Our Everest: keep the skills, lose the hardware

Now for our Everest, the claim that decides the framework's reach:

Communication skills learned from brain feedback during training persist at inference, after the EEG is removed.

We call this open-loop empathic transfer. The reasoning behind it is less hand-wavy than it might sound, and it rests on three things.
  • The setting already has a name in machine learning. The brain signal during training is privileged information (Vapnik and Vashist, 2009), an extra channel available to the teacher but not at test time. It has an established track record.
  • We have commercial precedent in-house. Arctop already ships models trained on paired content-and-neural data that then predict from content alone, the approach described in our empathic computing patent (Furman and Kwalwasser, 2021).
  • The architecture supports the switch natively. The same trained model runs closed-loop, reading real decoded scores, or open-loop, reading its own predictor's estimates through an identical interface. It isn't told which one it's getting.
The human analogy is simpler. Great communicators build their models of other minds through years of rich, embodied feedback. Then they write a book, and the skill survives on the page with no real-time feedback at all. Rich feedback during learning, internalized models at deployment. That's the shape of the claim. The paper distinguishes two strengths of the hypothesis. The merits and disadvantages of each can be debated separately.
  • Level 1, the conservative version. The model internalizes population-level regularities about what pacing, density, and tone tend to do to human cognition, and applies them from textual cues. This needs nothing beyond statistical learning.
  • Level 2, the aspirational version. The model learns to infer an individual reader's current state from conversation alone.
Level 1 alone would be a big deal. It would mean brain data from a modest training population could make models measurably better communicators for everyone, whether or not there's a sensor on their head when the conversation happens. Level 2 would be monumental: genuine machine theory of mind. The paper flags it as speculative and scientifically contested. The strongest objection, which the paper raises against itself, is that the model might simply learn to be nice (warmer, simpler, blander for everyone) and score well without adapting to anyone. We call this neurological sycophancy, because it's the same failure RLHF has with stated preferences, just displaced to brain-pleasure signals. The paper commits to five diagnostic tests for it, probing whether the trained model modulates complexity by context, corrects user errors even when correction costs comfort, and treats different negative emotional states differently instead of blanketing them all with warmth. And because a hypothesis this convenient deserves a fair chance to die, the paper pre-registers an experiment designed to kill it. Four conditions from the same base model: no post-training, RLHF, RLbF with live EEG, and (the one that matters) RLbF with the EEG off. Sixty completing participants, within-subjects, everyone wearing an active-recording EEG in every condition so nobody can tell which one they're in. The decisive comparison is open-loop RLbF against RLHF, and the RLHF arm should be read as a stand-in. Modern post-training already goes beyond it (preference optimization, AI feedback, constitutional rules), and the general claim is bigger than beating one acronym: a modern model trained with a brain in the loop should outperform the same model trained with the same recipe minus the brain. Better empathic quality with no brain signal at inference would be evidence that the training left durable skills behind. The sample sizes, the statistical tests, and what failure would look like are all specified in advance.

Reading brains, responsibly

We know how this sounds: an AI that reads your brain and adapts to your state before you're consciously aware of it yourself. The paper devotes its longest section to the problem, and two commitments define our answer. The first is a bright line between empathic and persuasive computing. A persuasive system uses knowledge of your state to drive you toward a target: a purchase, an opinion, another hour of engagement. An empathic system treats your state as input, not target. It changes its own behavior to fit where you are, and it doesn't steer you anywhere. That's why the application-layer weight defaults to zero, and it's why the paper's deployment checklist makes a published reward specification and formal verification of no-hidden-target blocking requirements. The checklist runs fourteen points in all, covering consent, user control (a hard off-switch included), and scheduled sessions without adaptation, so users' own communication resilience doesn't quietly atrophy. The second is data minimization as architecture rather than policy. Raw EEG is encrypted on your device and travels exactly one hop, to the decoder. Only the derived scores go further, stripped of everything else a brain signal might reveal. The paper is equally frank about what design cannot fix. A system that sees your cognitive state in real time knows things about the interaction that you don't, and transparency can shrink that asymmetry but never eliminate it. It also analyzes its own worst-case misuses by name, from engagement optimization to interrogation. Section 8 carries the full framework, including its mapping onto neurorights and emerging regulation (Ienca and Andorno, 2017; UNESCO, 2025). We treat that section as load-bearing. If the ethics fail, the framework fails, whatever the benchmarks say. That is the other Isaac's law, and it applies here.

What we're claiming, and what we're not

Read the paper and you'll notice the phrase this is a hypothesis over and over. That's deliberate. What the paper contributes:
  • A formal framework: the state space, the loop, the reward, with every design decision argued.
  • A production platform in Isaac, live in the Arctop apps, generating the aligned neuro-conversational data the field lacks.
  • A falsifiable central bet in open-loop transfer, with a pre-registered experiment built to test it honestly.
  • An ethical framework designed in from the start.
What it does not contain is empirical validation of RLbF training itself. There is no trained model yet and no experimental results. Every claim in the paper is labeled either as grounded in cited evidence or as pending the evaluation program it lays out. One idea here survives skepticism about any single component. Every previous alignment signal, from human preferences to AI feedback to constitutional rules, is a judgment about communication, made from outside, after the fact. Brain feedback is a measurement of communication, taken from inside, while it happens. Teaching machines to speak in ways brains actually find useful, rather than ways people report liking or ways engagement metrics reward, is a different objective than this field has ever optimized. Whether it works is now an empirical question, and that was the point of writing it all down: precisely enough that it can be proven wrong.

Read the full paper on Preprints.org, or explore the Isaac platform documentation on GitHub. Or, if you have MW75 Neuro headphones, join the loop yourself: the Arctop app is on Google Play and the App Store. If you think the central hypothesis is wrong, I'd like to hear why: DM me directly on LinkedIn.

Competing interests: Both authors are co-founders of and employed by Arctop Inc., which develops the brain-decoding technology and the Isaac platform described in this post.

Cortex, 154, 348-364. https://doi.org/10.1016/j.cortex.2022.06.005" }, { "html": "Al-Shargie, F., Kiguchi, M., Badruddin, N., Dass, S. C., Hani, A. F. M., and Tang, T. B. (2016). Mental stress assessment using simultaneous measurement of EEG and fNIRS. Biomedical Optics Express, 7(10), 3882-3898. https://doi.org/10.1364/BOE.7.003882" }, { "html": "Berretz, G., Packheiser, J., Wolf, O. T., and Ocklenburg, S. (2022). Acute stress increases left hemispheric activity measured via changes in frontal alpha asymmetries. iScience, 25(2), 103841. https://doi.org/10.1016/j.isci.2022.103841" }, { "html": "Borghini, G., Astolfi, L., Vecchiato, G., Mattia, D., and Babiloni, F. (2014). Measuring neurophysiological signals in aircraft pilots and car drivers for the assessment of mental workload, fatigue and drowsiness. Neuroscience & Biobehavioral Reviews, 44, 58-75. https://doi.org/10.1016/j.neubiorev.2012.10.003" }, { "html": "Cavanagh, J. F. and Frank, M. J. (2014). Frontal theta as a mechanism for cognitive control. Trends in Cognitive Sciences, 18(8), 414-421. https://doi.org/10.1016/j.tics.2014.04.012" }, { "html": "Cheng, M., Lee, C., Khadpe, P., Yu, S., Han, D., and Jurafsky, D. (2026). Sycophantic AI decreases prosocial intentions and promotes dependence. Science, 391(6792), aec8352." }, { "html": "Chikhi, S., Matton, N., and Blanchet, S. (2022). EEG power spectral measures of cognitive workload: A meta-analysis. Psychophysiology, 59(6), e14009. https://doi.org/10.1111/psyp.14009" }, { "html": "Coan, J. A. and Allen, J. J. B. (2004). Frontal EEG asymmetry as a moderator and mediator of emotion. Biological Psychology, 67(1-2), 7-50. https://doi.org/10.1016/j.biopsycho.2004.03.002" }, { "html": "Davidson, R. J., Ekman, P., Saron, C. D., Senulis, J. A., and Friesen, W. V. (1990). Approach-withdrawal and cerebral asymmetry: Emotional expression and brain physiology: I. Journal of Personality and Social Psychology, 58(2), 330-341. https://doi.org/10.1037/0022-3514.58.2.330" }, { "html": "Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. (2009). ImageNet: A large-scale hierarchical image database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 248-255. https://doi.org/10.1109/CVPR.2009.5206848" }, { "html": "Dietrich, A. (2004). Neurocognitive mechanisms underlying the experience of flow. Consciousness and Cognition, 13(4), 746-761. https://doi.org/10.1016/j.concog.2004.07.002" }, { "html": "Ding, N. and Simon, J. Z. (2012). Emergence of neural encoding of auditory objects while listening to competing speakers. Proceedings of the National Academy of Sciences, 109(29), 11854-11859. https://doi.org/10.1073/pnas.1205381109" }, { "html": "Ehrhardt, N. M., Fietz, J., Kopf-Beck, J., Kappelmann, N., and Brem, A.-K. (2022). Separating EEG correlates of stress: Cognitive effort, time pressure, and social-evaluative threat. European Journal of Neuroscience, 55(9-10), 2464-2473. https://doi.org/10.1111/ejn.15211" }, { "html": "Furman, D. and Kwalwasser, E. (2021). Empathic Computing System and Methods for Improved Human Interactions With Digital Content Experiences. US Patent Application US20210390366A1, Arctop LTD. https://patents.google.com/patent/US20210390366A1" }, { "html": "Geirnaert, S., Vandecappelle, S., Alickovic, E., de Cheveigné, A., Lalor, E., Meyer, B. T., Miran, S., Francart, T., and Bertrand, A. (2021). Electroencephalography-based auditory attention decoding: Toward neurosteered hearing devices. IEEE Signal Processing Magazine, 38(4), 89-102. https://doi.org/10.1109/MSP.2021.3075932" }, { "html": "Gevins, A., Smith, M. E., McEvoy, L., and Yu, D. (1997). High-resolution EEG mapping of cortical activation related to working memory: Effects of task difficulty, type of processing, and practice. Cerebral Cortex, 7(4), 374-385. https://doi.org/10.1093/cercor/7.4.374" }, { "html": "Giannakakis, G., Grigoriadis, D., Giannakaki, K., Simantiraki, O., Roniotis, A., and Tsiknakis, M. (2022). Review on psychological stress detection using biosignals. IEEE Transactions on Affective Computing, 13(1), 440-460. https://doi.org/10.1109/TAFFC.2019.2927337" }, { "html": "Harmon-Jones, E., Gable, P. A., and Peterson, C. K. (2010). The role of asymmetric frontal cortical activity in emotion-related phenomena: A review and update. Biological Psychology, 84(3), 451-462. https://doi.org/10.1016/j.biopsycho.2009.08.010" }, { "html": "Ienca, M. and Andorno, R. (2017). Towards new human rights in the age of neuroscience and neurotechnology. Life Sciences, Society and Policy, 13(1), 5. https://doi.org/10.1186/s40504-017-0050-1" }, { "html": "Katahira, K., Yamazaki, Y., Yamaoka, C., Ozaki, H., Nakagawa, S., and Nagata, N. (2018). EEG correlates of the flow state: A combination of increased frontal theta and moderate frontocentral alpha rhythm in the mental arithmetic task. Frontiers in Psychology, 9, 300. https://doi.org/10.3389/fpsyg.2018.00300" }, { "html": "Kay, E., Furman, D., Kogan, B., and Chiang, K.-J. (2026). Reinforcement Learning from Brain Feedback (RLbF) for Large Language Model (LLM) Improvement: Using and Evaluating Real-Time Neurophysiological Reward Signals for System Adaptation. Preprints.org. https://doi.org/10.20944/preprints202608.1274.v1" }, { "html": "Kerlin, J. R., Shahin, A. J., and Miller, L. M. (2010). Attentional gain control of ongoing cortical speech representations in a “cocktail party”. Journal of Neuroscience, 30(2), 620-628. https://doi.org/10.1523/JNEUROSCI.3631-09.2010" }, { "html": "Khoshnoud, S., Alvarez Igarzábal, F., and Wittmann, M. (2020). Peripheral-physiological and neural correlates of the flow experience while playing video games: A comprehensive review. PeerJ, 8, e10520. https://doi.org/10.7717/peerj.10520" }, { "html": "Klimesch, W. (1999). EEG alpha and theta oscillations reflect cognitive and memory performance: A review and analysis. Brain Research Reviews, 29(2-3), 169-195. https://doi.org/10.1016/S0165-0173(98)00056-3" }, { "html": "Kopito, R., Haruvi, A., Brande-Eilat, N., Kalev, S., Kay, E., and Furman, D. (2021). Brain-based Authentication: Towards A Scalable, Commercial Grade Solution Using Noninvasive Brain Signals. bioRxiv. https://doi.org/10.1101/2021.04.09.439244" }, { "html": "Krizhevsky, A., Sutskever, I., and Hinton, G. E. (2012). ImageNet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems, 25. https://proceedings.neurips.cc/paper/2012/hash/c399862d3b9d6b76c8436e924a68c45b-Abstract.html" }, { "html": "Lin, B., Guo, B., Zhuang, L., Zhang, D., and Wang, F. (2025). Neural oscillations predict flow experience. Cognitive Neurodynamics, 19(1), 1. https://doi.org/10.1007/s11571-024-10205-x" }, { "html": "Mesgarani, N. and Chang, E. F. (2012). Selective cortical representation of attended speaker in multi-talker speech perception. Nature, 485(7397), 233-236. https://doi.org/10.1038/nature11020" }, { "html": "Mirkovic, B., Debener, S., Jaeger, M., and De Vos, M. (2015). Decoding the attended speech stream with multi-channel EEG: Implications for online, daily-life applications. Journal of Neural Engineering, 12(4), 046007. https://doi.org/10.1088/1741-2560/12/4/046007" }, { "html": "Onton, J., Delorme, A., and Makeig, S. (2005). Frontal midline EEG dynamics during working memory. NeuroImage, 27(2), 341-356. https://doi.org/10.1016/j.neuroimage.2005.04.014" }, { "html": "O'Sullivan, J. A., Power, A. J., Mesgarani, N., Rajaram, S., Foxe, J. J., Shinn-Cunningham, B. G., Slaney, M., Shamma, S. A., and Lalor, E. C. (2015). Attentional selection in a cocktail party environment can be decoded from single-trial EEG. Cerebral Cortex, 25(7), 1697-1706. https://doi.org/10.1093/cercor/bht355" }, { "html": "Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., et al. (2022). Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35. https://arxiv.org/abs/2203.02155" }, { "html": "Palacios-García, I., Silva, J., Villena-González, M., Campos-Arteaga, G., Artigas-Vergara, C., Luarte, N., Rodríguez, E., and Bosman, C. A. (2021). Increase in beta power reflects attentional top-down modulation after psychosocial stress induction. Frontiers in Human Neuroscience, 15, 630813. https://doi.org/10.3389/fnhum.2021.630813" }, { "html": "Sweller, J. (1988). Cognitive load during problem solving: Effects on learning. Cognitive Science, 12(2), 257-285. https://doi.org/10.1207/s15516709cog1202_4" }, { "html": "Ulrich, M., Keller, J., and Grön, G. (2016). Neural signatures of experimentally induced flow experiences identified in a typical fMRI block design with BOLD imaging. Social Cognitive and Affective Neuroscience, 11(3), 496-507. https://doi.org/10.1093/scan/nsv133" }, { "html": "Ulrich, M., Keller, J., Hoenig, K., Waller, C., and Grön, G. (2014). Neural correlates of experimentally induced flow experiences. NeuroImage, 86, 194-202. https://doi.org/10.1016/j.neuroimage.2013.08.019" }, { "html": "UNESCO. (2025). Recommendation on the Ethics of Neurotechnology. United Nations Educational, Scientific and Cultural Organization. https://www.unesco.org/en/ethics-neurotech" }, { "html": "Vanhollebeke, G., De Smet, S., De Raedt, R., Baeken, C., van Mierlo, P., and Vanderhasselt, M.-A. (2022). The neural correlates of psychosocial stress: A systematic review and meta-analysis of spectral analysis EEG studies. Neurobiology of Stress, 18, 100452. https://doi.org/10.1016/j.ynstr.2022.100452" }, { "html": "Vapnik, V. and Vashist, A. (2009). A new learning paradigm: Learning using privileged information. Neural Networks, 22(5-6), 544-557. https://doi.org/10.1016/j.neunet.2009.06.042" } ]} /> --- # Why Consumer EEG Embedded Hardware Is the Next Big Platform: My Predictions as a CTO URL: https://arctop.com/deep-dives/consumer-eeg-hardware Published: 2025-03-28

Introduction: A New Wave of Technology in Everyday Life

I believe we're on the brink of a new platform revolution in consumer tech: brain sensor-embedded wearable devices. Specifically, EEG sensors will be embedded in devices at large scale, providing a platform for developing new services on, much like how GPS, accelerometers, and cameras went from novel extras to must-have sensors in every device. Why the excitement for a few new sensors? Because they unlock something fundamentally new and unprecedentedly powerful: the ability for our devices to sense cognitive state directly from the brain of users live in the moment. Fetching that class of emotional, cognitive data enables frictionless and powerful new input methods to computers, opening up a new frontier of possibilities. In this post, I'll explain why I predict consumer EEG hardware will be the next big platform, and how it will power smarter, more adaptive experiences in the very near future.

From GPS to Brainwaves: A Familiar Pattern

Brainwave-sensing hardware is poised to go mainstream by 2025–2026. As a CTO in this field, I'm seeing clear signs everywhere that what we predicted years ago is finally becoming reality. Hardware developers are energized – they're gearing up to embed miniature, dry EEG sensors into headphones, glasses, VR headsets, and more at scale – so app developers can build with next level capabilities, making App developers psyched also which will lead to an abundance of amazing experiences that will drive network effects of consumer adoption. Not long ago, you may recall, it was hard to imagine that every phone would have a GPS chip or a gyroscope. But as history shows, once a sensor becomes small and cheap enough to integrate en masse, it can rapidly turn into a platform for innovation. Smartphones today pack a suite of embedded sensors (accelerometers, compasses, gyroscopes, GPS, microphones, cameras, etc.) that enable countless applications across search, mobility, health, gaming, education and beyond. Developers ran with those capabilities – giving us turn-by-turn directions, fitness trackers, AR games, and social media filters that we now take for granted. I see EEG sensors following a similar trajectory. Early brainwave gadgets—like research-grade headsets or niche meditation bands—were isolated devices suited only to specific contexts. At worst, they were 'garbage-in-garbage-out' (“GIGO”) devices, technically capable of capturing EEG but with so much noise and unreliability that extracting meaningful data for computer interfaces or health insights was nearly impossible. But now high-quality EEG is breaking out of the lab, and the AI revolution in software is ready to embrace the new wave of EEG devices as optimized interfaces for daily actions like interactions with AI agents, fitness and health. Daily is the key here, like a toothbrush - most people will use the technology every day as part of their routine and it will get woven into much of life. We're now seeing brain sensors embedded directly into everyday consumer products — earbuds, headphones, AR/VR headsets, even wearable headbands and luxurious sleep masks. The leap from a specialty device to a common smartphone accessory is the game-changer around the corner. When EEG hardware is standard on consumer devices, it becomes a platform for developers to build on, just like GPS did for location-based apps. In other words, brain-computer interface tech won't be confined to research or medical implants any longer; it will be in the hands (and ears) of everyday users.

Why EEG Wearables Are Set to Go Mainstream by 2025

Several converging factors make me confident that 2025 is the tipping point for EEG wearables:
  • Maturing Technology: Consumer hardware has finally caught up to the vision. EEG sensors are now compact and power-efficient enough to tuck into normal-looking peripherals. For example, one newly released set of premium headphones from Master & Dynamic hides multiple EEG electrodes in the ear cushions, while looking and feeling like regular over-ear headphones. The difference is literally imperceptible to the wearer, but what the software can do for them if they are wearing the EEG one versus the non-embedded one is night and day.
    MW75-Neuro headset with EEG electrodes in the ear cushions"} sourceUrl="https://www.masterdynamic.com/products/mw75-neuro" sourceLabel="Master & Dynamic" /> At CES 2025 we also saw brain-sensing earbuds debuting from a few leading companies – sleek wireless buds that measure EEG from inside the ear canal and surrounding areas.
  • Developer & Industry Enthusiasm: As Arctop's CTO, I have the privilege of speaking with many hardware makers and product teams. The excitement in these conversations is palpable. Companies across the spectrum – from audio tech brands to AR/VR innovators – are actively at some stage of planning, prototyping or deploying EEG-enabled devices. We're talking smart headphone manufacturers adding health-sensing features, AR glasses startups looking to integrate EEG for context awareness and brain ID as passwordless login, and VR headset teams exploring cognitive inputs for mental health apps. Even major tech companies have signaled interest – industry trends show that big players are “increasingly exploring physiological sensors as user inputs for AR/VR.” AI companies also are assessing how and where BCI technology can be used to give them an edge, or at the least - to avoid strategic surprise. In short, the personal computing hardware industry is betting that brain-sensing is the next frontier, and the giants don't want to be disrupted and left behind.
  • User Demand for Deeper Insights: The timing is right from a user perspective too. Over the past few years, consumers have grown comfortable with wearables that track heart rate, sleep, stress, and more. There's a growing cultural emphasis on mindfulness, focus, and mental well-being. People are asking for tech that not only counts steps, but also helps them understand and improve their cognitive and emotional state. EEG data fits perfectly into this trend. Imagine being able to see when you are truly focused versus mentally fatigued, or to objectively measure how calm your mind is during meditation. Or even better, let your 'Calm' app measure how calm you are in real-time and do whatever it can to increase that calmness uniquely for you. That kind of insight was only available in labs – now it's on the cusp of being available on your headphones or earbuds at home. Early evidence of demand can be seen in the success of products like the Muse meditation headband and the interest around Emotiv's new EEG earbuds. Users are excited to get their brain “vitals” in the same way they've come to expect heart-rate or steps on their smartwatch. They're curious to track themselves for self-knowledge, and also for actionable "SOS" early and emergency warning signals for themselves, their parents, kids or friends. Basically they want to learn and grow with the help of their personal biometric data in a variety of contexts, like how Whoop and Oura users came to love their new devices and build upon what began as a health trend of people tracking their steps.
  • Better Algorithms & SDKs: Raw EEG signals are extremely complex, but here's the good news—developers don't need a neuroscience Ph.D. to make sense of them, just as they don't need a physics Ph.D. to implement GPS-based location features. Advances in machine learning and signal processing (including work by my team at Arctop) have made it much easier to interpret brainwave data in real time. We now have software that can detect patterns in EEG corresponding to recognizable mental states (cognitive workload, enjoyment, auditory attention, etc.) with surprising reliability. As an industry, we've finally created the abstraction layers needed to shield the gory neuroscience details from product developers and give them only what they need. (For example, our SDK at Arctop makes integrating cognitive state sensing plug-and-play, so developers can get meaningful metrics like “focus level” or “enjoyment level” derived from a device's EEG with just a few lines of code – no need to reinvent signal processing from scratch, let alone reinvent it for every compatible hardware device and platform). This means when EEG hardware lands in developers' hands, they can immediately start building cool apps and features that will be interoperable with multiple devices and application platforms instead of spending years researching brainwaves. The tooling is now at a point where cognitive signal integration is as straightforward as tapping into an accelerometer or camera feed. This is major for accelerating adoption.

Unlocking a New Dimension: Cognitive State Sensing

The most profound reason EEG hardware is exciting is that it unlocks a new dimension of user understanding. At consumer scale, that means fundamentally new experiences for humans around the globe on a daily basis. Our devices have always been good at sensing the external world for us or our physical state – location, motion, touch, sound, vision, heart rate, etc. But they've been blind to what's happening inside us. Inside that mush in our heads between our ears is a lot of electrical activity that is key to how we experience the world, to say the least. Finally that mush is being measured accurately and understood, and beyond that - being connected to computer understanding of user intention and perception. Consumer EEG, in other words, changes the computing paradigm. For the first time, mainstream gadgets will be able to capture aspects of our cognitive and emotional state in real time. Why is this a big deal? Because having access to the user's cognitive state lets devices behave in ways that are far more context-aware, personalized, and empathetic. I'm talking about software that adapts not only to your physical environment ie. where you are, but to you, the individual user, who you are in that very moment. This extra layer of understanding – sometimes called cognition-aware design, or human-centric – is poised to revolutionize user experience. It's like going from 2D black-and-white to 3D full spectrum color. Suddenly apps can respond to whether you're confused or confident, overwhelmed or bored, angry or delighted. Developers can design interactions that flex to the user's mental state on the fly which makes technology feel significantly more humane and responsive. In fact, we've already seen early research and products hint at what's possible: everything from personalized learning systems to assistive communication devices to emotion-adaptive games powered by real-time brain data and interpretations on top of that electrophysiological continuous flux. I believe that as this capability becomes widely available, we'll look back and wonder how our computers ever functioned without understanding our state of mind.

Smarter, More Adaptive Experiences Ahead

What kinds of new experiences can cognition-aware devices enable? The possibilities span many domains and again, similar to GPS-enabled location apps, the creativity of developers to create unpredictable viral hits cannot be underestimated. But let's explore a few clear and compelling use cases that developers and product teams are already dreaming up for user state-aware devices equipped with embedded EEG:
  • Adaptive Audio Headphones: Imagine your smart headphones detecting that your mind is wandering or your stress is spiking, and automatically adjusting the audio to help. If you're distracted at work, the headphones could dial up noise cancellation or switch to an instrumental playlist to improve your focus. If they sense you're stressed, they might play calming ambient sounds or gentle biofeedback tones to steady your nerves. The result is an adaptive audio experience that tunes itself to your mental state in real time.
  • Immersion-Aware AR/VR Headsets: Next-gen mixed reality headsets (think Vision Pro–style devices) will likely incorporate EEG to gauge your mental load and fatigue. Such a headset could detect when you're getting overwhelmed or tired during an intense VR session and dynamically adjust the experience for example through the immersion level. If cognitive fatigue is detected, the system might dim the brightness, simplify the visuals, or reduce the immersion intensity to maintain comfort. It could even suggest a short break or switch to a more passive content mode when your focus drops. Immersion level becomes fluid, automatically tuning up when you're engaged and scaling back when your brain needs a breather. This would keep extended AR/VR use comfortable and personalized, preventing tech burnout by respecting the user's cognitive limits.
  • Engagement-Adaptive Learning: One of the areas I'm most excited about is education. E-learning platforms and training apps can become dramatically more effective if they know the learner's engagement and cognitive state. With EEG sensors in a pair of study headphones or AR glasses, a learning app could detect if you're truly absorbing the material or if your attention is fading. If the system senses high engagement and flow, it could capitalize on that moment – maybe increase the challenge or speed through easier content. If it detects confusion or mental fatigue, it could pause to review the concept in a different way, or insert a short interactive to re-capture your attention. The pacing and difficulty adapt continuously to maximize the learner's information retention and minimize confusion. This kind of cognition-aware tutoring could help keep students out of the dreaded boredom or overload zones, optimizing learning efficiency. It's like having a tutor who can read your mind and personalize the lesson instantaneously.
These scenarios are just the tip of the iceberg. Virtually any experience that can benefit from understanding how the user is feeling or responding in the moment could tap into brain signals to become more adaptive. Gaming, for instance, might adjust difficulty based on a player's stress or focus. Fitness apps could use brain data to find the sweet spot between pushing you and not overstressing you. Even automotive interfaces might monitor driver alertness to improve safety. The common theme is that devices will no longer be oblivious to the user's mindstate – and that opens up an incredible design space for innovation.

Enabling Developers: Lowering the Barrier with Plug-and-Play Neuroscience

One question I often get is, “This sounds awesome, but how can our team actually implement brain-sensing features without neuroscience expertise?” This is where I gently highlight the role of platforms like ours. At Arctop, our mission has been to make brain-computer interface technology accessible to developers. In practice, that means we provide an SDK and APIs that handle the heavy lifting of EEG signal processing and interpretation. We've spent years decoding the language of the brain so that you, as a developer or product designer, don't have to. The goal is to make integrating cognitive signals as easy as any other sensor. You get high-level metrics (for example: “user's cognitive workload = 70%”) that your software can use to adapt the experience. Meanwhile, under the hood our platform is doing the complex work of filtering EEG noise, running machine learning models, and translating neural patterns into those useful insights. By abstracting away the neuroscience, we let product teams focus on what they do best – building creative applications and delightful user experiences. It's similar to how early smartphone app developers didn't need to know the physics of GPS satellites; they just called an API to get location and then built cool services on top of it. We envision EEG data working the same way: request cognitive state info, get a stream of values/events, and build with it. This plug-and-play approach is already helping developers experiment with cognition-aware features without a steep learning curve. I mention Arctop's role not to pat ourselves on the back, but to reassure the tech community that the tools are ready for this next big platform. The industry (including us and others) has been laying the groundwork so that when the new EEG-enabled hardware hits the market, developers can hit the ground running. We want to fast-forward to the creative part – seeing what amazing new applications you all will invent when devices truly understand users at the cognitive level.

Conclusion: Cognition-Aware Design is the Future

In my view, cognition-aware design represents the next evolution in human-computer interaction. We've made our devices context-aware about our external world; next they will become context-aware about us – our focus, enjoyment, interest, and more. Consumer EEG hardware is the key that unlocks this capability, and its impending mainstream arrival is why I'm so confident it's the next big platform. The years 2025–2026 will likely be remembered as the time when neurotechnology quietly slipped into everyone's daily gadgets. For developers, product teams, and forward-looking technologists, the message is clear: now is the time to start thinking about how brain-computer interface capabilities can enhance your products. The hardware is coming – in some cases, it's already here – and the demand for more intuitive, personalized, and responsive tech experiences is only growing. Early adopters who build user state-aware applications are going to shape the next generation of tech user experience, just like the first movers in mobile apps did with location services or accelerometer-based interactions. Investors and innovators take note as well: an ecosystem of EEG-enabled wearables and apps is emerging, with opportunities to lead in new market categories (EEG wearables, cognition-aware apps, adaptive audio, brain-health tech, and so on). At Arctop, we're thrilled to be enabling this trend in our own way, but it's the broader industry and developer community that will truly bring it to life. I've focused my career on brain-computer interfaces because I genuinely believe this technology will make technology more human-centric. When our devices can sense and respect our cognitive state, they can collaborate with us more intelligently. They can nudge us to be our best, or help us when we're not at our best. They can make experiences feel magically tailored in the moment, which is ultimately what users crave – technology that “gets” them. Thank you for reading this deep dive into the future of consumer EEG hardware. I hope it inspires you to think about how this technology will transform the way we interact with technology.

Further reading: Reinforcement Learning from Brain Feedback (RLbF) for Large Language Model Improvement takes cognition-aware design down to the model layer — using decoded EEG as a reward signal when post-training large language models (preprint: 10.20944/preprints202608.1274.v1).

--- # Memory Decoding at Home: Mapping Inter-Subject Neural Synchrony to Episodic Memory Performance URL: https://arctop.com/deep-dives/memory-decoding-from-eeg-at-home Published: 2024-07-15 This post is meant as an informal, quick transmission of a technical discovery to introduce our novel algorithm (FCCA) for mapping inter-subject neural synchrony to memory function. Fuller description of methods and discussion to follow in an upcoming version. The authors would like to thank collaborators Erez Simony, Aia Haruvi, Shai Kalev, Ronen Kopito for their contributions.

Abstract

Neural synchrony, similarities in brain activity across individuals, offers valuable insights into emotional and cognitive processing. It can also be used, under certain conditions, to infer subjective states of consciousness1. Objective metrics of similarities between people's brain data, or inter-subject correlations (ISC), we map here to dynamics of memory performance using a novel algorithm that builds upon the framework developed in correlated components analysis (CorrCA), generating a new higher-order analysis layer that obtains superior performance in memory decoding from EEG data compared to all other known methods. We call the method feature-based CorrCA (FCCA), and find it enables a robust, accurate memory analysis from people at home, in non-laboratory environments, involved in everyday tasks. By selectively deriving and using particular EEG activity features relevant to cognitive and emotional processes reflected in neural synchrony, FCCA serves as a powerful tool for investigation of the neural basis of shared experiences and highlights the nuance needed for accurate memory decoding from EEG. Validated on two models of consumer-grade EEG recording devices, our generalizable approach offers new possibilities for studying collective experiences in various contexts, from education to entertainment, and contributes to the study and understanding of social cognition and memory. Episodic memory in particular has been an active area of research for modern neuroscience and there is a wealth of published findings in the literature relating to how physiological measures of brain activity during information encoding map to behaviorally demonstrable memory performance in the future. (Lee 2022)(Rugg, 2007, 2013), (Simony 2016), (Summerfield 2006). Studying hippocampus-cortex interaction during movie watching has been one fruitful experimental direction for the field. Along this line of inquiry we report on a naturalistic neuroscience study we conducted involving 60 adult participants distributed across the United States. Participants measured their EEG brain signal using a consumer wearable headband with sensors embedded while watching 38 short video clips and a 27-minute episode of the TV show 'Curb Your Enthusiasm' and days later were quizzed through multiple choice tests on aspects of the content to investigate whether FCCA can help elucidate relationships between memory formation and neural synchrony. We found that ISC dynamics derived by our FCCA algorithm can effectively capture memorability level dynamics while individuals experience the same stimulus.

Introduction

Earlier research has established a significant relationship between inter-subject neural synchrony and memory2,3. Both inter-brain synchrony and memory were separately found to be related to the emotional valence of events4–7. However, studies connecting the dynamics of these three processes to commonly experienced content are lacking. Traditional approaches have primarily relied on research-grade equipment and restrictive experimental paradigms, with limited research into the capabilities of consumer-grade EEG that are accessible for use at home nor the element of privacy and personal freedom that influences how participants experience TV shows at home versus in a social, focus group setting. This gap suggested to us the opportunity to develop an algorithm for populations in the real world experiencing the same content. Recent advancements in neuroimaging techniques provide for a detailed exploration of neural synchrony, and the emergence of hyperscanning, the neuroimaging of multiple individuals simultaneously, has proven the utility of such algorithms for research and medical applications 8,9. This progress has highlighted the usefulness of robust brain synchrony algorithms and especially those that are suitable for real-time analysis with consumer-grade neural activity measurement devices that not only function effectively with lower signal quality and fewer channels but additionally provide real-time feedback on neural synchrony in naturalistic settings outside of the hyper controlled laboratory10. The tantalizing possibility of measuring memory formation effectively through these techniques led us to develop new algorithms for EEG that would satisfy these criteria. Correlated Components Analysis (CorrCA) has emerged as a promising method for the study of inter-brain neural synchrony, enabling the extraction of maximally correlated components across individuals' neural recordings11. Projections from the hippocampus to the prefrontal region, for example, we hypothesized to show up with CorrCA while being missed by other methods. Indeed, CorrCA-derived inter-subject correlations (ISC) were found repeatedly to be related to the common cognitive aspects of experience across individuals in response to stimuli, such as engagement 11,12 interest13 and memorability3. CorrCA, as it was originally designed to utilize EEG channels data as an input, allows the computation of scalp projections, which provide valuable information about the spatial distribution of correlated neural activity across the scalp 11,14. Compared to the more familiar Canonical Correlation Analysis (CCA), CorrCA is designed to find shared components across multiple datasets or subjects while maximizing the correlation of these components across the datasets. This approach also enables multiple datasets as inputs from different subjects or experimental runs, outputting a set of components that are maximally correlated across all input datasets. A key assumption of CorrCA we accept is that there are underlying components generated by the human brain that are shared across datasets. In other words, that because the source of this data is the human brain we accept as a sufficient common element of each data source to validly map across participants by, while still allowing for individual variations in the cognitive and emotional components meaning of that data. A benefit of methods in the CorrCA class is that we can interpret the shared components across all datasets. In this study, we present an enhanced approach to CorrCA, called features-based Correlated Components Analysis (FCCA). This method examines the synchrony level between individuals based on the dynamics of EEG activity features rather than on denoised neural activity itself, thus focusing on synchrony originating in the activity properties rather than physical localization. That is the main contribution of this paper: FCCA by orienting towards relationships in activity properties and not physical space, is able to overcome spatial limitations of EEG to extract MRI level insights for a fraction of the cost, and far more frequently since the technique is possible from data recorded at home. Our results demonstrate the validity of FCCA, and its potential to derive insights from the dynamics of neural synchrony between subjects about engagement, emotional positivity, and memory levels. The proposed comprehensive algorithmic approach carries significance for contexts in which there is a need for robust real-time estimation of group experience in naturalistic settings, using consumer-grade recording devices and, we believe, has far reaching implications for various fields. Where memory is an important factor, such as education, entertainment, and clinical applications, FCCA can inform strategies for enhancing learning, creating memorable content, and advancing understanding of emotional engagement dynamics and memory formation in a variety of contexts.

Methods

We conducted two experiments using commercially available EEG headbands with participants who watched the same set of video clips from home. The first experiment involved short video clips viewing with subsequent emotional ratings and a memory test. The second experiment included full episode viewing followed by a memory test a few days later.

Participants

All participants had normal or corrected-to-normal vision, provided written informed consent, and received payment for their participation. Exclusion criteria included the use of medication that might influence the experiment or the presence of neurological or psychiatric conditions. Inclusion criteria required participants to have normal vision or vision corrected to normal with contact lenses. In the first experiment, there were 32 participants (mean age = 36, SD = 8.25, 16 females). In the second experiment, 27 participants (age 33, SD = 4 years, 13 females). For the memory test related to the American TV episode in experiment 2, the subjects were distributed approximately evenly across the five major regions of the continental United States (Northeast, Southwest, West, Southeast, and Midwest) and all spoke English as their native language.

Data Acquisition

In both experiments, participants viewed the videos within a mobile Android app ("Arctop" by Arctop, Inc.), while their electrical brain activity was recorded. EEG activity was recorded using Muse headbands with EEG sensors embedded. For Experiment 1, the Muse 2 headband was used, while the Muse S headband (Interaxon Inc.) was used for Experiment 2. Both devices are portable, noninvasive electroencephalography (EEG) devices with a sampling rate of 256 Hz, containing four dry EEG sensors located on the scalp at two frontal channels (AF7 and AF8) and two temporal channels positioned behind the ears (TP9 and TP10), with a reference channel at Fpz. The first experiment was conducted in an office building with a support staff available to assist the participant with the device and the experimental protocol, in the second experiment each participant was mailed an Arctop technology kit at home that included headphones (Sony), brain signal measuring headband, and a tablet computer (Samsung) with the Arctop app installed. The headbands were put on by the participants themselves, with the assistance of a Quality Assurance (QA) screen that started before each session. The QA showed the participants, in real-time, the data' quality, easily directing them to adjust the headband properly for optimal signal quality. In both experiments, recording was performed while the participants were alone in a quiet room.
The experiment was composed of two sessions, separated by a break. During these sessions each subject watched overall 38 short clips, each ranging from 32 to 100 seconds in length, with a median duration of 59 seconds. The video clips collection was composed of clips with positive (e.g. babies laughing), negative (e.g. An excerpt from "The Champ"15), and neutral affective valence (e.g. moving abstract shapes). In a second experimental day, adapted from the study of Hasson et al.16, participants viewed a 27-minute-long episode from the English-speaking television sitcom Curb Your Enthusiasm (Season 1, Episode 7: "AAMCO" by Larry David). The episode depicted various independent events, such as a dinner party and a minor car accident. A list of the content in table.

Emotional Ranking

In the first experiment, after watching each clip, participants ranked their emotional responses across eight emotions using a slider button in the Arctop app: enjoyment, interest, happiness, dislike, boredom, stress, relaxation, and sadness. We calculated the average score per emotion for each stimulus.

Memory Assessment

In both experiments, subjects weren't aware that they would perform a memory test. Instead, after the end of the experiment they were notified that there was an additional online task that they could participate in, for which they would receive additional compensation. They received the memory test via email 3 days after the video-watching session. We verified that the email was received, and the participants were instructed to perform the test in a quiet room. In both experiments, the test was administered via a Google Form. Each question included a still frame from the relevant time interval as a visual cue and provided three alternative forced-choice answers. Participants were instructed to choose one answer and rate their confidence level on a Likert scale ranging from 1 to 6. Memory performance was derived by calculating the percentage of correct answers per question across subjects. For the first experiment, the memory test was composed of 3 questions about events from each of four of the clips (out of the 38 they watched), i.e. 12 questions overall. For the second experiment, the memory test consisted of 69 questions about the episode's narrative. The questions were designed so that the events they addressed were separated by approximately 20 seconds, and each question could be answered using information from the relevant segment of the episode alone.

EEG-Based Between-Subjects Synchrony Estimation

Data Preprocessing
EEG data were preprocessed to remove artifacts and filtered to retain relevant frequency bands. A band-pass filter (0.5–48 Hz) was applied to each channel together with a notch filter (either 50 Hz or 60 Hz) to remove line noise. During the performed tasks, 1.5 s of EEG data segments were extracted from the filtered signal using a sliding window with a stride of 100 ms. Data was excluded based on dropped packets of EEG data length relative to the stimulus, which occurred due to Bluetooth disconnections: deviation of more than a second was set as the threshold for exclusion in both experiments. Besides excluding data based on the recording duration threshold, recordings that had more than 5% missing values were excluded. Other data curation approaches were taken to exclude invalid data while maintaining the sample sufficient for each stimulus according to the heuristic of EEG-based CorrCA calculations are valid for eleven participants and more13. Preprocessing was completed with feature extraction, where from each EEG segment (epoch) a total of 54 features were extracted according to the current version of Arctop software platform processing (July 2024), resulting in a matrix of epochs X features X subjects created per stimulus to be used in subsequent FCCA analysis routines.
Correlated Components Analysis (CorrCA)
To examine the stimulus-evoked neural activity synchrony level between subjects, we utilized CorrCA, which was applied to yield the ISC of the neural responses 3,12,13,13,17. Correlated component analysis extracts projections of the data with maximal correlation by finding linear combinations of the EEG data or features dynamics over time. As described in detail by Cohen and Parra3, the procedure involves the calculation of the pooled between-subject cross-covariance,
and the pooled within-subject covariance,
where
measures the cross-covariance of all the time-series data sources in subject k with all sources in subject l. When CorrCA is implemented on the EEG data, these sources refer to the collected neural activity data, such as EEG-electrodes data. When calculating it on the features data, instead of a matrix of time by electrodes, the matrix is of time by features. Vector Xk(t) represents the scalp voltages, or features dynamics, at time t in subject k, and, x̄k, their mean value in time. Obtaining the ISC involves simultaneously diagonalizing the pooled covariance and the cross-covariance of the two data sets. The linear components that achieve this can be obtained as the solutions of a generalized eigenvalue equation:
Solving for
leads to ISC per component, as the component projections that capture the largest correlation between subjects are the eigenvectors
of the matrix
with the strongest eigenvalues. Thus, the strength of correlation in the ith component is obtained by:
High ISC is obtained when the dynamics of either EEG or features are similar across subjects. Prior to computing eigenvectors, the pooled within-subject correlation matrix is regularized to improve robustness to outliers using shrinkage.
ISC Noise Level Estimation
To assess the ISC difference from noise level, i.e. remove time-dependent intersubject entrainment while maintaining each subject's temporal and spatial correlation, we implemented a phase scrambling method 18,19. This method involves selecting a random phase offset for each subject and circularly shifting all data points along the time dimension by this offset. Specifically, for each subject, we first computed a feature matrix for the full EEG length. The circular shift was then applied along the time axis of this matrix, followed by segmenting the matrix according to event labels. The circular shift was applied to each event separately and ISC was then calculated over the circularly shifted data, either over the entire event (for overall ISC) or by windows of 30 samples (for ISC dynamics). This process was repeated one hundred times, with a new circular shift applied. ISC computed in each iteration, yielding a distribution that estimated the noise for each stimulus.
ISC Significance Test
The circular-shuffle-based distribution approach allows the generation of a null distribution while maintaining the inherent structure of the original data. The ISC p-value was computed per component. The p-value of the overall ISC was calculated as 1 minus the cumulative distribution of all circular-shift ISCs for a specific component. The dynamic p-value was computed similarly for each time point. To validate the approach of applying CorrCA on the dynamics of the feature as opposed to the EEG signals, each ISC output of both approaches was compared relative to the noise level. The p-value relative to this distribution was calculated per stimulus, and ISCs whose calculated p-value was smaller than 5% were considered significantly different from noise. We estimated the fraction of significant ISCs for the same set of stimuli, depending on the CorrCA approach.
ISC Measures
Overall ISC
For the relation of the inter-brain synchrony with the emotional ranking, we utilized the ISC for each stimulus, due to the rankings' being referred to the entire stimulus. The ISC we refer to here is the eigenvalue of the eigenvalue problem mentioned above. We extracted this measure of the first three components.
ISC Scores
When looking into the relation of memory levels relative to the ISC level of specific events or scenes in the videos, we calculated the ISC dynamics, as it is expected different scenes evoke varying levels of ISC. We compute the correlations in a time-resolved fashion11 by employing a sliding window with a 3-s duration with a shift of the window occurring every second. In the first experiment, we calculated the ISC scores of each of the three first components in the time windows which correspond to the parts of the video where the memory questions were asked, and their relation to the memory accuracy. The ISC dynamics were averaged within the time frame that corresponds to the event that the question refers to. The mean ISC per event was normalized by subtracting the error rate of that time frame, as calculated by the aforementioned noise estimation procedure. Note that when the average ISC in the given time window is higher than the error rate, the obtained value is positive, and vice versa. For the episode analysis, the ISC values were also calculated in partially overlapping time windows as in the first experiment. Subjects with missing values were removed from analysis at each time window. The shuffle-based noise level per time window was subtracted from the ISC value per window, to obtain a normalized ISC curve along the episode. The normalized ISC curve within each memory test question's relevant time frame was averaged, yielding an ISC measure per question. Both the obtained ISC and the memory performance curves underwent convolution with a moving average of 0.5 to 3 seconds running window.
Factor Analysis for Emotions Ranking
Factor analysis was performed to uncover latent structures within the emotional rankings and their relation to ISC. As a preliminary step towards conducting the factor analysis, we examined the multicollinearity among the average emotional ranking by stimulus using the Variance Inflation Factor (VIF), to ensure that the factors extracted are not influenced by redundant or highly correlated variables.
Variance Inflation Factor (VIF) Analysis
High multicollinearity, indicating redundancy, was identified for emotions receiving scores between 5 to 10. We found that enjoyment and happiness ranking exhibited moderate to high multicollinearity (VIF values of 9.45 and 5.71, respectively). To address this issue, we combined enjoyment and happiness into a single composite variable. Following this step, the revised VIF values indicate that the issue of multicollinearity has been effectively handled, with all variables displaying low to moderate multicollinearity (minimum VIF of 1.68 and Maximum VIF of 4.11) and, therefore, low redundancy.
Determining the Number of Factors
To determine the number of factors for the factor analysis, we employed the scree plot method (named for the plot's resemblance to scree rock cliffs in nature) and the Kaiser criterion (Supplementary Figure 1). Based on this analysis, we proceeded with three factors for our factor analysis. The resulting three factors in the factor analysis (after inverting their signs for clarity) can be interpreted as "Emotional Positivity," "Arousal," and "Engagement", respectively (Supplementary Figure 2). Factor 1, "Emotional Positivity," is characterized by high positive loadings for the enjoyment/happiness composite (0.581) and relaxation (0.47), and high negative loadings for stress (-0.84), dislike (-0.62), and sadness (-0.60). This factor represents a clear contrast between positive and negative emotions. Factor 2, "Arousal," includes high positive loadings for dislike (0.48) and stress (0.43), and high negative loadings for sadness (-0.74). This factor captures emotions related to reactivity, stress, and sadness, and thus can be interpreted as capturing arousal-related emotions. Notably, the second component's ISC was found to be positively correlated with memory accuracy. Factor 3, "Engagement," is defined by high positive loadings for interest (0.77) and enjoyment/happiness composite (0.59), and high negative loadings for boredom (-0.84). This factor represents engagement versus disengagement. These results further demonstrate the validity and efficacy of the FCCA approach in yielding robust inter-subject correlations (ISC) as the first three components revealed their distinct relationships with cognitive processes, suggesting that ISC derived from FCCA is informative about both emotion and memory processes in response to content.
Relation of ISC to Memory Scores
In Experiment 1, we investigated the relationship between memory accuracy for specific mini-scenes and ISC scores. ISC dynamics were extracted for each stimulus, and the mean ISC within the time frames corresponding to the mini-scenes was calculated and normalized by subtracting the error rate. For each of the first three components, the Shapiro-Wilk test was performed to choose between Pearson's and Spearman's correlation coefficients. The p-values were adjusted using the Bonferroni correction method. For Experiment 2, we assessed memory accuracy and ISC dynamics using a range of convolution window sizes (0.5 to 3 seconds). The correlation between memory accuracy and ISC dynamics was calculated, with the Shapiro-Wilk test performed to assess normality and choose the appropriate correlation coefficient. Corresponding p-values were adjusted for multiple comparisons using the Bonferroni correction method.

Statistical Analysis

All statistical analyses were performed via Python (version 3.12.3; "Statsmodels" and "SciPy" libraries). Pearson or Spearman correlation coefficients were chosen based on the results of the Shapiro-Wilk test for normality. The p-values that are reported refer to the hypothesis test whose null hypothesis is that two samples do not correlate. All p-values were adjusted for multiple comparisons using the Bonferroni correction method.

Results

In the current study, frontal EEG activity was recorded using a Muse headband while subjects watched video stimuli in two different designs (Muse 2 for the first design and Muse S for the second). In an offline-analysis, the data was preprocessed, EEG features were extracted and neural synchrony was calculated. In the first design, each subject watched 38 short clips (ranging from a minimum of 32 seconds to a maximum of 100 seconds, a median of 59 seconds) and was asked to rank eight emotions (enjoyment, interest, happiness, dislike, boredom, stress, relaxation, and sadness; Figure 1a) after each clip. Three days later, they performed a memory test to estimate their accuracy in recalling mini-scenes from several clips viewed during the experiment (Figure 1b). In the second design (Figure 1c), participants viewed a 27-minute-long episode and performed a memory test on mini-scenes from the episode. In both designs, the subjects watched the clips and took the memory tests alone. They were unaware that they would be taking a memory test beforehand. See the designs' description elaboration in methods.

Features-Based CorrCA Approach Validation

We calculated ISC for each stimulus based on the EEG features. The ISC distributions of the first three components are plotted in Figure 2a. To assess their difference in the stimulus-specific noise level, we created a "circular-shuffles"-based ISC distribution (i.e. randomly shifting in time each subject's signals to remove time-dependent intersubject entrainment). We compared the discriminability from noise for ISC values derived from performing CorrCA (i.e. calculated for the dynamics of the EEG signals) and FCCA (i.e. calculated based on the EEG-derived features (see methods for elaborated explanation). We found that the implementation of FCCA led to ISC scores that were robust and were significant relative to shuffled data (for 33 out of 38 stimuli; Figure 2b), to a greater extent relative to EEG-based CorrCA (for 1 out of 38 stimuli; Figure 2c).
33 out of 38 stimuli, or 86.8%."} />
1 out of 38 stimuli, or 2.6%."} />

Mapping Emotional Space Relative to FCCA-Derived ISCs

Having established that FCCA yields extremely robust ISCs that are approximately 30 times more informative than the nearest methods, we turned to examining the relationship between emotional responses and the ISC values obtained by our approach. Multiple emotional responses for each stimulus were ranked by each subject (see Figure 1a). Emotional responses are inherently complex and multidimensional, involving a variety of distinct but interrelated feelings (for example, feelings of happiness and enjoyment might co-occur but not be fully redundant, as might feelings of sadness and dislike). Factor analysis enables us to reduce this complexity, and examine whether the emotional rankings share common factors that are linked to the ISC of each component separately. Due to its ability to uncover latent structures that represent the combined effects of multiple emotions, Factor analysis can potentially reveal underlying patterns that may not be apparent in pairwise comparisons. Following eliminating redundancy and determining the number of factors in a data-driven manner (see methods), factor analysis was performed for 3 factors. The resulting factors are interpreted as "Emotional Positivity," "Arousal," and "Engagement", respectively (Supplementary Figure 2).
Visualization for the factor loadings of emotional rankings across three factors (i.e. the contribution of each emotion to the identified factors): "Emotional Positivity," "Arousal," and "Engagement." Each cell represents the loading of an emotion on a specific factor, with the color scale indicating the magnitude and direction of the loading.

Emotional Factors Correlations with ISC

To examine the relation between each of the emotional factors and the ISC of the first three components, we performed a correlation analysis and conducted Bonferroni correction for multiple comparisons. The results indicate that only the "Engagement" factor has a significant positive correlation with the ISC of the 1st component (r= 0.48, p=0.006; Figure 3a), suggesting that a higher score for this component is associated with higher engagement. "Emotional Positivity" and "Arousal" did not show significant correlations with ISC of Component 1 (r=-0.16, r=0.05; Figure 3a). The ISC of component 2 was negatively correlated with the "Emotional Positivity" factor (r=-0.44, p=0.015; Figure 3b), but not to the other two factors. Interestingly, the ISC of the third component was not found to be significantly correlated to any of the factors (Figure 3c). The "Engagement" factor showed a significant positive correlation with ISC of the 1st component (r=0.48, **p=0.006). "Emotional Positivity" and "Arousal" factors were not significantly correlated with ISC of Component 1. The ISC of component 2 was negatively correlated with the "Emotional Positivity" factor (r=-0.44, *p=0.015). ISC of the 3rd component showed no significant correlations with any of the factors.

Memory Decoding from FCCA-Derived ISCs

Short Clips

We investigated the relationship between the memorability of short scenes from a selection of short videos, and the ISC score (n=32, design no. 1; Figure 5a). Specifically, we examined the ISC scores of each of the three first components in the time windows corresponding to the parts of the video where the memory questions were asked, and their relation to the memory accuracy. To that end, we extracted the ISC dynamics for each stimulus, calculated the mean ISC within the time frame that corresponds to the mini-scene, and normalized it by subtracting the error rate of that time frame (see methods; example in Figure 4a). Note that when the average ISC in the given time window is higher than the error rate, the obtained value is positive, and vice versa. A statistical analysis was performed to examine the correlation between memory accuracy and the ISC scores. After applying Bonferroni correction, we found that the only significant result was for the second component's ISC, which had a positive Pearson correlation (r = 0.711, p = 0.0284) between memory accuracy and the ISC scores. For the first component, the analysis revealed a positive Spearman correlation (r = 0.663), however, this correlation was marginally significant (p = 0.06; Supplementary Figure 3a). For the third component, the analysis showed a weak Pearson correlation (r = 0.135, p = 0.676; Supplementary Figure 3b) between memory accuracy and the ISC scores. This correlation was not statistically significant before the Bonferroni correction. Overall, these results suggest a specific relationship between the ISC scores of the second component and the memory accuracy for the corresponding video segments.
Example of ISC dynamics and the shuffle-based error rate, and the time frame corresponding to the events the questions relate to. Shows significant positive Pearson correlation between memory accuracy and ISC scores for the second component (r=0.711, **p=0.0284).

Curb Your Enthusiasm Episode

We wanted to further examine the relation between memory decoding capabilities and FCCA-generated ISC. First, we wanted to examine the reproducibility of the specific relation to the second component's ISC that we observed in the first experiment. Second, to gain a more thorough perspective of the relation between the dynamics of these two processes, by having dozens of mini-scenes within a longer stimulus, as opposed to a diverse set where differences in brain states and latent variables of attention and emotion may encode differently due to juxtapositions in time alone. This approach allows us to characterize the relation between fluctuations in the time series pair. Lastly, since in the first experiment, the relation between the memory accuracy and the first component's ISC was marginally significant, we sought to further examine the nature of this relation. To that end, we adopted a previously established fMRI experimental procedure, which was designed to examine the relation between memory accuracy and the level of inter-subject synchrony, along the time course of a longer stimulus16. Specifically, we obtained memory accuracy for questions about 69 independent mini-scenes from a 35.5-minute "Curb Your Enthusiasm" episode (n=11; design no. 2) and extracted the ISC scores that corresponded to their time frames. This episode was chosen for its preponderance of independent mini-scenes or events that are disjointed from one another and thus do not necessarily contain information about one or another, which allows examining memory accuracy for specific time-points along the stimulus in a constructive way to relate it to the ISC level. The TV show stimulus was much longer, approximately a half hour, and accordingly, the relevant events were spread over a duration of several seconds. We accounted for the possible shift in time between the ISC and the memory level time series in a data-driven manner by performing a correlation analysis of the ISC score during the mini-scenes timing to the accuracy in answering these questions, while smoothing both time series with a moving average. Specifically, the correlation between the memory level and the ISC score dynamics was calculated for a range of running average convolution window sizes ranging from 0.5 seconds to 3 seconds, in increments of 0.5 seconds. A significance test was performed to each convolution value and the corresponding p-values were adjusted for multiple comparisons using the Bonferroni correction method. We found that the relation between the memory accuracy and the second component's ISC score dynamic was replicated in this design. The correlation in the first convolution window size of 0.5 seconds was found to be positive, however it was marginally significant, (r= 0.45, p=0.09). After applying the Bonferroni correction, we found that all other comparisons were statistically significant (for 1 second p<0.05, for 1.5 and above, p<0.0001). This result further corroborates the relation of the 2nd component's ISC and emphasizes its stability. Figure 5a depicts the surprisingly stable and robust relation between the ISC and memory accuracy of the 2.5-second convolution.
After corroborating the relation of memory accuracy to the 2nd component's ISC, we turned to examine its relation to the 1st component's ISC, as it was marginally significant in the first experiment. We conducted the same set of analyses and statistical tests and found no significant, or marginally significant results (Figure 5c). This finding strengthens the specificity of the memory accuracy relation of the FCCA-derived ISC of the second component.

Discussion

Our study introduces a novel approach to CorrCA, termed FCCA, which examines synchrony based on EEG activity features. We found this method to be highly robust, compatible with consumer-grade EEG devices, and allows obtaining insights into the common emotional and cognitive experiences based on the neural synchrony between subjects. After establishing that on our data set, FCCA yields results that differ from noise at a higher rate than CorrCA, we found that the FCCA approach revealed distinct relationships between ISC of the first two components and the video-clips' emotional ranking by the subjects. Our analysis showed that the "Engagement" emotional factor strongly associated with the first component's ISC. This result aligns with previous research on the relationship between neural synchrony and engagement 3,19,20, and therefore corroborates the validity of the FCCA approach. Importantly, our study extends beyond engagement to explore the relationship between neural synchrony, as measured by FCCA, and memory which is known to be largely housed anotomically in deep structures that are closed-fields to EEG signal. We found that the relation between ISC and memory was specific to the second component. This result was evident both when examining the memory for selected events taken from several short video clips and also when these events were taken from the time span of a longer episode. Our findings are in agreement with the previous findings of Cohen and Parra3, who demonstrated the relation between the first 3 components' ISC relation to memory level when applying the CorrCA analysis to EEG data. Therefore, the ISC dynamics derived by our algorithm can effectively capture memorability level dynamics while individuals experience the same stimulus. The second component's overall ISC and the "Emotional Positivity" factor were negatively correlated. This finding aligns with the study of Nummenmaa et al.21, reporting an association between negative valence and increased inter-brain synchrony in the emotion-processing network, including prefrontal regions. Together with our result that the second component's ISC level corresponds to the memorability dynamics along the stimulus, our findings point to an inverse relationship between emotional positivity and memory accuracy. This finding is in agreement with evidence suggesting that sad or negative stimuli are more memorable6. Future research could explore how the dynamics of emotional valence along the stimulus correspond to the ISC dynamics, and how different emotional valences in context interact with memory formation. As FCCA provides a comprehensive framework that is suitable for either offline or real-time tracking of emotional and cognitive processes simultaneously, future studies could also examine how different types of stimuli, ranging from educational materials to various forms of entertainment, affect experience. This approach may facilitate creation of personalized content in education and entertainment, potentially informing the design of engaging and memorable content. Finally, future studies could extend the use of this robust algorithm to the hyperscanning research field, by enabling tracking emotions, and memory processes based on large groups neural dynamics. Previous findings that relate the synchrony level calculated by CorrCA to individual traits, such as age22 and familiarity or expertise with respect to the displayed visual stimuli23 (Goldberg, 2014). Therefore, given the potential applicability of FCCA for naturalistic settings, it may be implemented for identifying divergence from a common synchronized neural response, allowing the detection of abnormal patterns in individuals' traits, such as emotional reactivity or cognitive processing. This avenue may be highly relevant in educational and clinical settings, in the diagnosis and treatment of disorders affecting social cognition or memory. In conclusion, our study demonstrates that our proposed approach is a powerful and versatile tool for investigating neural synchrony. By providing a nuanced understanding of how emotional responses and memory processes are reflected in neural synchrony, FCCA opens new pathways for studying common patterns within and between collective experiences across various contexts. The ability of FCCA to yield meaningful results across different kinds of consumer-grade recording devices significantly enhances its generalizability, expanding the scope of real-time tracking of neural synchrony in naturalistic settings that is possible through a variety of headwear from earbuds with temporally located EEG sensors to AR/VR devices with frontally located EEG sensors.

Supplementary

Eigenvalues of the principal components in descending order for emotional factors. The Kaiser criterion, which retains factors with eigenvalues greater than 1, supported retaining three factors.

References

1. Nummenmaa, L., Lahnakoski, J. M. & Glerean, E. Sharing the social world via intersubject neural synchronization. Curr. Opin. Psychol. 24, 7–14 (2018). 2. Hasson, U. & Frith, C. D. Mirroring and beyond: coupled dynamics as a generalized framework for modeling social interactions. Philos. Trans. R. Soc. B Biol. Sci. 371, 20150366 (2016). 3. Cohen, S. S. & Parra, L. C. Memorable Audiovisual Narratives Synchronize Sensory and Supramodal Neural Responses. eNeuro 3, (2016). 4. Kensinger, E. A. Remembering the Details: Effects of Emotion. Emot. Rev. 1, 99–113 (2009). 5. Santamaria, L. et al. Emotional valence modulates the topology of the parent-infant inter-brain network. NeuroImage 207, 116341 (2020). 6. Cunningham, T. J., Leal, S. L., Yassa, M. A. & Payne, J. D. Post-encoding stress enhances mnemonic discrimination of negative stimuli. Learn. Mem. 25, 611–619 (2018). 7. McGaugh, J. L. The amygdala modulates the consolidation of memories of emotionally arousing experiences. Annu. Rev. Neurosci. 27, 1–28 (2004). 8. Czeszumski, A. et al. Hyperscanning: A Valid Method to Study Neural Inter-brain Underpinnings of Social Interaction. Front. Hum. Neurosci. 14, (2020). 9. Dumas, G., Nadel, J., Soussignan, R., Martinerie, J. & Garnero, L. Inter-Brain Synchronization during Social Interaction. PLoS ONE 5, e12166 (2010). 10. Nazneen, T. et al. Recent Trends in Non-invasive Neural Recording Based Brain-to-Brain Synchrony Analysis on Multidisciplinary Human Interactions for Understanding Brain Dynamics: A Systematic Review. Front. Comput. Neurosci. 16, 875282 (2022). 11. Dmochowski, J. P., Sajda, P., Dias, J. & Parra, L. C. Correlated Components of Ongoing EEG Point to Emotionally Laden Attention – A Possible Marker of Engagement? Front. Hum. Neurosci. 6, (2012). 12. Poulsen, A. T., Kamronn, S., Dmochowski, J., Parra, L. C. & Hansen, L. K. EEG in the classroom: Synchronized neural recordings during video presentation. Sci. Rep. 7, 43916 (2017). 13. Dmochowski, J. P. et al. Audience preferences are predicted by temporal reliability of neural processing. Nat. Commun. 5, 4567 (2014). 14. Truong, N. C. D. et al. Mapping and understanding of correlated electroencephalogram (EEG) responses to the newsvendor problem. Sci. Rep. 12, 13800 (2022). 15. Gross, J. J. & Levenson, R. W. Emotion elicitation using films. Cogn. Emot. 9, 87–108 (1995). 16. Hasson, U., Furman, O., Clark, D., Dudai, Y. & Davachi, L. Enhanced Intersubject Correlations during Movie Viewing Correlate with Successful Episodic Encoding. Neuron 57, 452–462 (2008). 17. Cohen, S. S. et al. Neural engagement with online educational videos predicts learning performance for individual students. Neurobiol. Learn. Mem. 155, 60–64 (2018). 18. Prichard, D. & Theiler, J. Generating surrogate data for time series with several simultaneously measured variables. Phys. Rev. Lett. 73, 951–954 (1994). 19. Parra, L. C., Haufe, S. & Dmochowski, J. P. Correlated Components Analysis - Extracting Reliable Dimensions in Multivariate Data. Neurons Behav. Data Anal. Theory 2, (2019). 20. Dikker, S. et al. Brain-to-Brain Synchrony Tracks Real-World Dynamic Group Interactions in the Classroom. Curr. Biol. 27, 1375–1380 (2017). 21. Nummenmaa, L. et al. Emotions promote social interaction by synchronizing brain activity across individuals. Proc. Natl. Acad. Sci. 109, 9599–9604 (2012). 22. Petroni, A. et al. The Variability of Neural Responses to Naturalistic Videos Change with Age and Sex. eNeuro 5, ENEURO.0244-17.2017 (2018). 23. Madsen, J., Margulis, E. H., Simchy-Gross, R. & Parra, L. C. Music synchronizes brainwaves across listeners with strong effects of repetition, familiarity and training. Sci. Rep. 9, 3576 (2019). 24. Goldberg, H., Preminger, S., & Malach, R. (2014). The emotion-action link? Naturalistic emotional stimuli preferentially activate the human dorsal visual stream. Neuroimage, 84, 254-264. 25. Lee, H., Chen, J. Predicting memory from the network structure of naturalistic events. Nat Commun 13, 4235 (2022). 26. Rugg, M. D. & Curran, T. Event-related potentials and recognition memory. Trends Cogn. Sci. 11, 251–257 (2007). 27. Rugg, M. D. & Vilberg, K. L. Brain networks underlying episodic memory retrieval. Curr. Opin. Neurobiol. 23, 255–260 (2013). 28. Simony, E. et al. Dynamic reconfiguration of the default mode network during narrative comprehension. Nat. Commun. 7, 12141 (2016). 29. Goldberg, I. I., Harel, M. & Malach, R. When the brain loses its self: prefrontal inactivation during sensorimotor processing. Neuron 50, 329–339 (2006). 30. Summerfield, C. et al. Neocortical connectivity during episodic memory formation. PLoS Biol. 4, e128 (2006). --- # Five Levels of Explanation (Part II): How Brain-Computer Interfaces Work URL: https://arctop.com/deep-dives/how-bci-works-part-2 Published: 2023-12-29 This is a continuation of our previous post where we explain brain-computer interfaces in five levels of difficulty, following WIRED's style. We recommend you first read Part 1 for levels 1 through 3. Now let's dive into levels 4 and 5. 

Level 4: Grad Student

With the previous background knowledge, as a 'graduate student' for the purpose of this blog post's structure, you might wonder what the key research questions in the BCI field are. What are challenges that are valuable and may be solvable in the next 3-5 years? In other words, what could be worthwhile questions to work on for a Masters of Ph.D. All three pillars of BCI, hardware, algorithms and applications, have ripe questions here at the close of 2023. Below we go more into some of these below, in a word these are hardware challenges, which abound in the size, weight, power, cost of BCI devices; software challenges where the overarching mission of understanding the brain is paramount (and possibly an impossible quest), and in applications the challenge of effectively achieving the user's goal for use, i.e. quality, is all that matters. All in all combine to create the value of BCI.

Hardware: SWaP-C Challenges to Mainstream BCI

Historically, computer technology has gotten smaller and smaller. The almighty 'miniaturization' principle. Though it may not sound profound, it is, and this trend is one of the reasons why BCI technology is now becoming available to people for everyday use. Electronic technology keeps getting smaller and smaller mainly because smaller devices fit into our lifestyles better. The computer industry is now regularly etching precise, detailed designs on the order of nanometers into silicon chips so our phones and computers can be lighter while also being more powerful: to emphasize — nanometers with precision — a supremely amazing feat. It's fair to say that as a species we have been fairly obsessed with miniaturization of our technology and very successful in scratching that itch. An extraordinary amount of engineering imagination and efforts have gone into shrinking components and making the already small still smaller, and the trend continues. The computer scientist Gordon Bell found that miniaturization is actually happening in regular intervals over time — generating new classes of computers in the process that, in his words: “…bring with them new markets, ecosystems, and — most importantly — new types of users.”
There is a natural combining that is currently underway with BCI technology being embedded in headphones, earbuds and AR/VR form factors converging to the human head and globally opening the era of BCI in consumer electronics. The electronics for amplification, filtering, digitization, communication underpinning these new capabilities have been consistently miniaturized, but brain sensors have been more constrained due to the biophysical properties of conductive ionic currents on the skin where sensor surface area is correlated with signal-to-noise-ratio (SNR). As sensors become integrated with existing technology and head-wear form factors the size won't matter much since it will be small enough to be functional invisible to the user.  Devices that are barely visible to the human eye already are designed to fit seamlessly into, and around, our bodies; the progression trends straightly towards the nanoscale. And already, these implanted sensors have allowed humans and computers to interact with each other directly. These new types of symbiotic relationships transform not only daily lives, but our entire species in the process; it's as if one day we will all grow another limb or develop, theoretically, a new sense. According to Bell, we are approaching the apex of small electronics. With that convergence to the minimum scale for electronics there may then be inversion: with technology going inward and contorting itself in new dimensions. As in, into our bodies — implants that seamlessly, for example, measure from the brain with a Young's Modulus matching jiggle, so the implant does not shear and damage tissue as it naturally jiggles about in daily life. The key research challenge in this pillar is how to make BCI sensors and devices easy to use, comfortable, affordable, unobtrusive, and reliable for long-term monitoring in daily life. In the past one to two decades, there have been efforts in both academic and industry and significant advancement in both invasive and non-invasive BCI technology. On the research side, continual breakthroughs have been made in materials of sensors that are biocompatible, flexible, and can form stable contact with skin - in the form of tattoos or fabrics for example. Another important direction is how to miniaturize the sensors and devices such that they are easy to put on by the users themselves, comfortable to wear, and can last for a long time, all the while recording reliable signals from the brain with sufficient spatial coverage.
Rogers Research Group."} /> We now see more companies tackling this particular challenge, delivering BCI products in the form of earbuds, headphones, eyewear, or that integrate with a VR headset or a hearing aid. Lastly, it's important to point out the breakthrough in invasive technology. Research labs and companies are pushing the boundary that maximizes the number of sensors and their spatial coverage of implanted devices while minimizing the risk and costs of surgical operations and long-term use. These are highly active research and development areas that will likely have many breakthroughs in the next 5 years. One framework for thinking about where to focus as a graduate student in a hardware field is the acronym: SWaP-Cs. Which stands for Size, Weight, Power, Cost — the goal is to decrease all of these variables. There's a dual meaning to it also: Style, Washability, Prestige, Comfort. Perhaps the areas that will advance BCI most today are in these alternative SWaP Cs where sexy, cool and comfortable meet in headwear products like glasses or earrings, hats, beanies and bindis, all supreme expressions of what a BCI can be. What will BCI devices look like in the future? Inspired by Mark Weiser’s view, we believe BCI will disappear into the fabric of everyday life and be indistinguishable from it. At Arctop we think of the technology as being an extension of cognition – of our everyday thinking, expanded and extended, embodied in the environment and technologies around it - without being felt as a technology. That's how seamless it should be.  Without the capability of remote sensing at a distance away from the brain activities, BCI sensors will need to be around, on, or inside the head. With such constraints, there is limited real estate of the head where people can (and are willing to) wear a device for a long time. This is why we are already seeing and will see even more in the near future, that BCI technology being miniaturized and integrated into existing eyewear or headwear, including eyeglasses, headphones, earbuds, hearing aids, headbands, helmets, and VR / AR headsets is the trend. 
link)."} /> Another trend is invasive technology - making implants smaller, safer, and more affordable. For now, the risk of invasive technology may only be justified by clinical or medical use for those in need, and the technology development still needs to go through rigorous clinical validation and regulatory processes to de-risk the adverse effects in long-term use. But in the long run, when it reaches a similar risk level as replacing a missing tooth, in exchange for unlocking greater human abilities, would you be willing to do so? We might be.
Interactive electronic content delivery in coordination with rapid decoding of brain activity. (Furman, Kwalwasser, 2023)."} />

Software: Algorithms for Decoding Brain Data and Infrastructure

Empathic Computing System and Methods for Improved Human Interactions With Digital Content Experiences. Information hierarchy showing how multimodal data and priors feed into an application environment. As population data grows, algorithm capabilities will too. (Furman, Kwalwasser, 2021)"} /> The decoding algorithms of BCI technology have mainly used digital signal processing and statistical machine learning techniques. Artificial neural networks have not worked so great traditionally for real neural network activity, but with recent breakthroughs in artificial intelligence (specifically deep learning methods) for text, image, and speech recognition, there has been an outpour of publications applying deep-learning methods to biosignals. But deep-learning has not been a magic wand that magically solves the recognition problem for brain activities yet. Why? The key challenges lay in both the unique data and algorithms. On the data side, unlike text and images, brain data has limited numbers of public, large-scale datasets for training models and benchmarking performance on. These datasets often have their own data acquisition hardware and idiosyncratic data collection protocols which introduce more variability. There is also a lack of standards on metrics and criteria for controlling signal quality and consensus for data cleaning methods despite wide consensus on the need for cleaning the always noisy signal. Lastly, limited "ground truth" labels are available and they often have low temporal resolution with high variability (e.g. disagreement even between “experts”). But in the last decade, significant efforts have gone into addressing these challenges and we are optimistic that these are likely solvable in the near future. For example, some BCI research communities and non-profit organizations have started to publish their datasets (OpenNeuro) and open-source codebase (NeuroTechX) to establish and follow a standard for data and label formats, for example 'EEG-BIDS' (Pernet et al 2019). Effective methods for handling artifacts and noises in EEG in the real-world setting have been proposed and evaluated (Chang et al 2019), and additional data modalities are being used (e.g. behaviors from phones, camera, physiological signals) to provide rich context and automated labels for the EEG. With these advances, we may be able to reach a critical point with enough data and labels that can drive the breakthrough in algorithm development. On the algorithm side, for readers who are AI researchers and developers wanting to transfer AI breakthroughs from other domains and solve the brain’s grand challenge, there are a few unique challenges for brain signals. First is the “context” or labels during which brain signals are collected. Unlike an image of a dog where the ground truth is unambiguous and universal, the ground truth for a period of brain signal can be noisy and subjective. There is not, for example, consensus about how many emotions humans can feel or the best way to establish the timing of when they are being felt. Hence approaches like self-supervised learning and multi-modal data for automated label generation are rising and necessary.  The second challenge is that the brain activities are always changing. Coming from sources throughout the brain, that overlap and transient signals that propagate in complex ways, there is a lot to analyze. Interestingly, brain activities seem to transition from a stationary state to another, nonstationary state at unbound timescales, like progression of sleep stages or a sequence of thoughts. But these brain states do not have fixed time intervals. Learning the “unit” of brain states, like learning the “vocabulary” of speech, may be the key to reduce the complexity of the brain decoding challenge and relax the data requirements. By transforming the problem from decoding time-series data to sequence data (e.g. leverage methods like brain-state modeling, Hsu et al 2018, 2022 or speech-to-unit translation) dimensions can be reduced and patterns can be modeled more effectively from first principles, obviating the need for massive data sets thay require more parsing. The third challenge is the “human factor” that changes the BCI performance from day to day or people-to-people due to differences in brain anatomy, sensor locations, or users’ states like attention, emotion, and motivation (Lotte et al 2013). To tackle this challenge, transfer learning techniques are needed to use data and pre-trained models from other days, other users, or even other devices to facilitate the “re-calibration” of your model (Chiang et al 2021). Adaptive learning is also being used to automatically, and continuously adapt the model to the user. 

Applications

BNCI Horizon 2020 (Brunner et al 2015), one of the highly influential BCI projects, presented five categories of BCI use cases: replace, restore, improve, enhance, and research. Here we propose a mental framework to understand the utility of BCI applications and to inspire fellow and future BCI pioneers to build upon.
BCI applications started off from “replacing body ability” (bottom left in the Utility Map) for paralyzed patients. With advancements in invasive technology, particularly implants and decoding capability, we will see clinical and medical-use BCI achieve higher accuracy, reliability, and degree of freedom of control for more diverse populations with disabilities. This includes finer movement control and speech decoding that would significantly increase the communication bandwidth and speed. BCI then expands toward “restoring brain ability” (upper left in the Map of BCI Applications) for patients with neurological disorders by coupling the brain wave to provide real-time, closed-loop stimulation for therapeutic interventions. BCI coupled with electrical or magnetic stimulation techniques, either non-invasively (TMS, tACS, tDCS, taVNS) or invasively (DBS, VNS or FES), have already been used or entered into clinical trials for treating a variety of neurological or mental disorders. Auditory stimulation in sync with slow-wave brain activities in sleep may improve sleep and potentially prevent cognitive decline (Zeller et al 2023). Visual feedback (e.g. play/stop a video clip) can be provided to incentivize users to achieve an ideal brain state for improving cognitive functions in people with ADHD or Schizophrenia (Singh et al 2020). Now we are seeing the trend in development and applications toward improving current abilities and unlocking new abilities (bottom and upper right in the Utility Map). Passive-BCI applications (Zander and Kothe, 2011) have been used in day-to-day settings for healthy populations in scenarios like learning in a classroom, working in the office or at home, driving a car, navigating an airplane, and playing and integrating in virtual-reality games. We will explore this topic further in Level 5. To sum up, with increasingly comfortable and miniaturized headwear for brain-sensing, the accumulation of data and labels, breakthroughs in artificial intelligence systems and computing capability for decoding, we will see BCI technology flourish and provide a better quality of life for us.

Level 5: Expert

What is the trend in BCI technology in the next 5-10 years and beyond? What are some grand challenges in BCI that may require breakthroughs from other fields? In this section, we invite you, as an expert in your own field, to collaborate and help unlock the future of BCI.

Scientific Understanding of the Brain

"Complete understanding of the human brain" is the grand challenge. It is so grand that almost certainly we will not arrive at a satisfactory result in this lifetime, one ought to walk humbly when approaching this mountain with its peak disappearing into the clouds. Like an ant learning the latest mathematical model of how ant societies work, our understanding may ultimately be limited by our vocabulary no matter how fluent we become. The goal can be pursued incrementally and eventually the mountain of knowledge needed might be scaled. We will know we've arrived when all neurological diseases, neurodegenerative disorders, mental illness and anything negative related to the brain does not exist anymore– since if we have complete understanding of it, it will mean we have complete ability to modify it. Before getting lost out on a philosophical or bioethics branch here lets opine back on the side of neuroscience and the grand challenge of the field: understanding the brain to the best of our abilities. The phenomena observed and documented by neuroscientific instruments seem only scratching the surface. Particularly, three research areas might have a direct impact on the future of BCI: (1) understanding the mechanism that gives rise to the complex and dynamic cognitive functions and mental states in humans. What are the neural mechanisms and cognitive processes that embody our subjective experience of feelings, thoughts, or other mental states? Without the bridges between neuroscience, cognitive science, and psychology, we won't be able to quantitatively measure the various mental states. How many emotions do we have after all? To measure them all we need to know. (2) advancement in tools to record and stimulate the brain with high spatial and temporal resolution - this will not only advance our scientific understanding of the brain but also provide BCI developers with the right tools to decode and modulate the desired brain functions. New tools lead to new rules, as the saying goes. And circularly, new rules to new tools. (3) using BCIs as a research tool itself, serving as a synthetic nervous system, that helps advance our understanding of neuronal, functional, and psychological changes in response to real-time, dynamical, adaptive feedback. Such understanding can create wondrous values, for example, to treat neurological or psychiatric disorders, find personalized methods to optimize our learning, promote our brain and mental health, and effectively co-evolve with the rapidly advancing artificial intelligent personal agents.

BCI Software Platform: from Algorithms to Mechanisms

Naturalistic interactions with computer applications is one primary goal of BCIs. Many of the most popular BCI paradigms focused on decoding algorithms and artificially connected the outputs for other forms of control as a proof-of-concept demonstration. These often are not natural for users and the interactions they create with applications can be unintuitive: flickering lights or flashing bars on a screen lead to a word being selected, or imagining opening or closing a hand steers a wheelchair. Many of these paradigms overlook the human in the loop and principles of usability that other personal computing areas. The connection between decoding algorithms and forms of interactive feedback or control, which together we refer to as the “Mechanism”, requires thoughtful and user-centered design. For a BCI to be useful and used on a daily basis, it needs to be comfortable to the extent of becoming unnoticeable. Not only physically, but psychologically as well in terms of the methods it employs to interact with the user. Here we'll offer a few unique BCI control and interaction mechanisms.
Actions mechanism can be produced by a user actively doing something and expecting fast and accurate outcome, like silent speech for communication or imagery movement for object control. Intent mechanism is a more subtle and natural form of control where a user wants something to happen by naturally changing what they focus on, prefer over, or dislike. States mechanism does not require any mental efforts and is a more subliminal, passive measure that reflects user emotions and general mental conditions. Responses mechanism also does not require active control but measures a user’s natural mental reaction to external stimuli or feedback, like seeing and correcting a mistake. Traditional human-computer interfaces only provide Actions mechanisms, such as pressing an icon with one's finger to open an app. It requires active effort to move the finger and press, and one would expect the app to open instantly every time it is pressed. Traditional BCI has followed this broader computing trend and pursued Action mechanisms. However, BCI has the unique ability to unlock other types of control and interaction mechanisms like Intent, States, and Responses in addition. These mechanisms require less mental efforts from users, and because of the more implicit control mechanisms, users do not have the same high expectation on performance compared to Actions. A successful BCI will in this way expand the command repertoire to provide various mechanisms that are intuitive to users, similar to how touchscreens and computer mice were instantly familiar and easy to start using by most people right away when they came out.  The ultimate solution likely involves a suite of mechanisms that inter-relate and interact with each other while driving external actions. Much like the brain itself is arranged, the aspects of cognition that connect to applications through a BCI likely need to be hierarchically arranged and segmented to work best.

What are the Killer Applications of BCI?

1. New Communications

Full conversational speech and full control over a computer are the goals for personal electronics generally when it comes to communications. Hands-free, eyes-free, voice-free, gesture-free commands that will work for patients and physically-able people alike. Pure intention and intuitive interactions easily working every time. In the first 50 years of BCI research, most applications focused on active control within medical, clinical, and research ecosystems to allow people who are paralyzed a new way to communicate or control a device that increases their independence. The noble goals and efforts in this arena have been mostly for helping patient populations where 80% control over a cursor to someone who is locked-in is immediately life changing. Continuing along this line of development, it is a feasible goal of the field to free every single locked-in patient and that, we believe, is achievable in the next decade with noninvasive methods. If a person has functional brain activity, they should be able to communicate basic commands at the least. This is a humanitarian objective of the field and needs to be achieved. Making locked-in a thing of the past remains a powerful driver for many in the field. For some it is the only litmus test worth evaluating progress by. From another perspective though, for most people, a cursor that only works 80% of the time is simply not good enough and the dominant BCI paradigms of the existing research feel strained and uncomfortable to use since they were designed for the most desperate of cases. In the past decade, a new ecosystem has been emerging towards a more natural use of BCIs where there is no need for active control. This demand on the user is lifted and interaction is more seamless. Easier connection paradigms makes BCI more accessible to average consumers. While implanted BCI systems might be needed to communicated at conversational speeds, there has been success in noninvasive systems that is accelerating, with adjacent advances in generative AI helping to power the progress.

2. Skills Learning

There are already killer applications within learning, mostly tied to how fast the same skills can be taught with a BCI and without. For example, when you are learning in an online course, an add-on BCI can accelerate your learning by sensing that you are bored and giving you harder materials; or sense that you are overwhelmed and slow down with more examples; tracking to optimal performance for people individually according to Yerkes-Dodson-like performance curves (Yerkes, Dodson, 1908). A driving trainer with add-on BCI can sense you feel stressed and modify its driving behavior to help you feel comfortable and in a more extreme example, an airline pilot can train in difficult and dangerous conditions that simulate their state of mind more effectively. Or imagine having a personal athletic trainer who can give you prompt notice when you seem to be distracted - to regain control of your attention at a key moment of an exercise. The keyword here is "action" - BCIs can take actions and turn insights from the brain into timely feedback that can effectively improve users' ability to learn material.

3. Adaptive Experiences

The third killer application is a personalized, adaptive experience directed by BCI. In gaming this is most immediate: a virtual hand with fingers you can control (Furman et al 2016) and use as easily as your own, an entire virtual body, an emotionally connected avatar. Gaming content, audio content, video content, all might be able to adaptively be tuned to individual users to improve their experience.  For example, audio content can be played at variable speeds for podcasts adapted to the user in the moment, and playlists can be customized to achieve an effect like increasing user focus (Haruvi et al 2022). Arctop technology has already shown some success in this area, being used to continuously measure a person's focus level while they listen to audio content. This enables adaptive playlists that reliably increase and sustain attention which is helpful for studying, working, exercising, and many other essential human tasks that rely on focus. In today's attention economy, information streams are increasingly rapid and bite-sized, with distractions abounding. Technologies that help maximize and maintain focus states can be invaluable. Using BCI technology, applications could adjust experiences to theoretically achieve any brain state aside from focus though, and that is perhaps the most 'killer' part of this application. By anchoring applications behaviors to objective measures of user cognition, a system of instant feedback becomes available that has profound implications of what is achievable for a user in concert with their BCI. 

Bioethics, Privacy & Security

Concerns about data ownership, data access rights, and data privacy abound in BCI. Data may be processed on devices (edge computing), locally, with user controlled data access rules and only sparse features (SDK) being sent out encrypted to the Internet and Cloud or local network. Or it could be processed like it's the cyber wild west with unrestricted data access rules over real-time and historic data.  Markets, leading companies and government regulation will likely be the greatest determiners of how these issues evolve. Across the world it will be interesting to see if there is consensus, or if divergent attitudes shape BCI technology in different directions. For now all we can do is build to the best principles possible, and design for privacy and the world we want to see.   BCI data is sensitive by nature since it contains information about a person’s identity (Kopito et al 2021), health and their real-time mental status so it needs to be treated accordingly. Through the cognitive and affective state data BCIs process, much of what people consider the most fundamental parts of themselves are exposed and because of that, to say the least, the field is ripe with ethical, security, and privacy issues that invite contributions from experts. Open questions in the field include how brain data is measured. For instance does the device clearly label that brain sensors are embedded? People at minimum should know if their brain data is being processed it seems. If they consent for it to be, the next layer is how is it used by applications? At Arctop we take a user privacy-centered approach, but this is not yet the prevailing model as increasingly cloud-based approaches are used for business models based on data being shared or accessed by affiliates. Some companies ask users to 'donate' their data, some are less straightforward. The dramatic differences such technology and data architecture decisions have on the end user's mental privacy and personal rights writ large cannot be overemphasized. BCI as a class of technology thus raises unique, complex issues. These involve human agency questions as well, since actions that BCIs perform must be treated as an extension of the person's own. Unless of course, the BCI made a mistake in decoding what the user wanted. For example if a prosthetic arm punches someone, who is to blame if the user denies that they made the BCI punch intentionally? At what point does a crime become a crime is another corner one may end up in doing work on the ethics in this space – does law enforcement need brain data to prevent drunk driving or a crime of passion? The reader can go ahead and fill in the blanks here in terms of other sci-fi tropes and subplots.  There is also a significant moral and societal question of BCI "haves" and "have-nots" and who is entitled to access the technology and for what purpose, since access to BCI technology could profoundly differentiate populations from one another and create positive and negative feedback loops across cultures if one has access to technology that accelerates learning and improves health that the other does not. (Bavelier et al 2019).

Toward a General BCI: Interface, Interaction, Intelligence

Gao et al 2021."} /> An interesting model is proposed by Gao et al 2021: that generalized BCI technology would evolve through three stages: interface, interaction, and intelligence. The field of BCI has transformed from “interfacing”, one-way brain-to-computer control, to “interaction”, two-way co-adaptation for both human brain and computer. This transition highlights the importance of the “Write” or “Encoding" path, compared to the conventional “Read” or “Decoding” direction. The “Write” path could use interactive auditory or visual feedback for adaptive learning experience in cognitive augmentation or skill learning, or it could use direct electrical stimulation as neuromodulation therapies or neurorehabilitation, as described in Level 4 Application. Since learning is a master tool we believe that the killer applications of read and write technology will be in the area of teaching ourselves new things not just faster, but more memorably and effectively. The era of super-human AI “agents” and as they become the new “computer”, the final stage of generalized BCI technology will be “collaborative intelligence” – a seamless integration and collaboration of human intelligence (HI) and artificial intelligence (AI) where learning is not needed in many instances since the partnership is coupled so tightly. HI is better at understanding, reasoning, generalizing, empathizing, and goal-setting, while AI is better at perceiving, memorizing, computing, interpreting, and achieving specific tasks. AI can couple a user's goal-directed intents (e.g. attention, preference) as reward functions in its reinforcement learning to align goals. AI can integrate a user's states (e.g. emotions, stress) to jointly make personalized, situational decisions in perfect harmony with goals aligned.

The Why: Building A General BCI

A General BCI is one that works for any human immediately out of the box with perfect decoding of all the elements and dimensions of a person's Cognition. It is a system that has mapped the human experiential space and has a unified informational model of how human's feel and understand things - a symbiotic relationship where the BCI does everything it can to maximize the user's quality of life. At scale, General BCI systems in use by a population of people through the same approach should be able to maximize the quality of life of the community as a whole. By making quality of life a quantified, data driven metric to be optimized by a General BCI the way can be charted towards improvement for anyone. Outgrowing certain human limitations and suffering with improved emotional regulation, reversing cognitive decline, maintaining memory and healthy forgetting, downloading skills into your brain to speak a new language. All are in the province of a General BCI, which by necessity requires leaps forward in imagination and understanding of the brain, human experience, and the environment in which both operate. In the recent book "The End of Reality," writer Jonathan Taplin raises the provocative point of the philosophy of the Greek Epicurus, which he ascribes to being the opposite of what 'Big Tech' aims for today in that for Epicurus the three elements of a happy life were unaddressable with modern technologies:
  1. The company of good friends
  2. The freedom and autonomy to enjoy meaningful work
  3. An 'examined life'
We would argue to the contrary that these Epicurean values are actually best addressed with a BCI – not other tech or by 'Big Tech' per se, we agree, but specifically BCI because of how it works – yet another example of how this technology is in a unique class. Perhaps instead of new communications, skills learning and adaptive experiences, those three elements above should be the anchors for BCI.  Company of good friends can be supported well by an empathic application powered by BCI for match making and scheduling and reviewing the goodness of the friend fit with objective data plus the patience and organization skill to maintain and cultivate friendships over time. Freedom and autonomy to work is about communication, control, about not being stopped from pursuing purposeful work regardless of where one is or who they are. And an 'examined life' is another word we feel for quantified self- the tech trend for which BCI is a type of apotheosis, since it allows the most high granularity examination of one's body and brain. So it may be that with BCI applied right, even the greatest Luddites among us may be won over. To create the ultimate General BCI that helps humans everywhere live a higher quality of life, at Arctop we are focusing on the software platform for decoding and building it in a hardware-agnostic way to be as universal in connective applications as possible. Quality of life for humans everywhere is our north star and to enable as many people to enjoy a high quality of life we believe the brain, and BCI technology, are the master tools. We can't do it alone since so much is required in the technological and societal stack to bring BCI out widely. Together we are at a unique time in human history with BCI, AI, and VR technologies knitted together by an ever more powerful compute that is near miraculously bringing us all into the General BCI era. With Arctop software and developer tools we aim to accelerate that arrival. Sometimes it's worth taking a moment to marvel at how far humans have come. Here at the end of 2023, the end of this post, we are grateful you're here reading and we are reflective on the distance run. But mostly we are looking ahead — excited to release a new product early in the new year that takes a leap forward to that goal of quality of life for all. If you want a sneak peak at our baby General BCI, reach out to us and we'll do our best to connect!

Postscript: the empathic, closed-loop BCI sketched above is now a formal framework. See Reinforcement Learning from Brain Feedback (RLbF) for Large Language Model Improvement, which turns decoded cognitive state into a continuous reward signal for post-training large language models (preprint: 10.20944/preprints202608.1274.v1).

Trends in cognitive sciences, 25(8), pp.671-684."},{"html":"Brunner, C., Birbaumer, N., Blankertz, B., Guger, C., Kübler, A., Mattia, D., Millán, J.D.R., Miralles, F., Nijholt, A., Opisso, E. and Ramsey, N., 2015. BNCI Horizon 2020: towards a roadmap for the BCI community. Brain-computer interfaces, 2(1), pp.1-10."},{"html":"Chang, C.Y., Hsu, S.H., Pion-Tonachini, L. and Jung, T.P., 2019. Evaluation of artifact subspace reconstruction for automatic artifact components removal in multi-channel EEG recordings. IEEE Transactions on Biomedical Engineering, 67(4), pp.1114-1121."},{"html":"Hsu, S.H., Pion-Tonachini, L., Palmer, J., Miyakoshi, M., Makeig, S. and Jung, T.P., 2018. Modeling brain dynamic state changes with adaptive mixture independent component analysis. NeuroImage, 183, pp.47-61."},{"html":"Hsu, S.H., Lin, Y., Onton, J., Jung, T.P. and Makeig, S., 2022. Unsupervised learning of brain state dynamics during emotion imagination using high-density EEG. NeuroImage, 249, p.118873."},{"html":"Chiang, K.J., Wei, C.S., Nakanishi, M. and Jung, T.P., 2021. Boosting template-based SSVEP decoding by cross-domain transfer learning. Journal of Neural Engineering, 18(1), p.016002."},{"html":"Singh, F., Shu, I.W., Hsu, S.H., Link, P., Pineda, J.A. and Granholm, E., 2020. Modulation of frontal gamma oscillations improves working memory in schizophrenia. NeuroImage: Clinical, 27, p.102339."},{"html":"Zeller, C.J., Züst, M.A., Wunderlin, M., Nissen, C. and Klöppel, S., 2023. The promise of portable remote auditory stimulation tools to enhance slow‐wave sleep and prevent cognitive decline. Journal of sleep research, p.e13818."},{"html":"Zander, T.O. and Kothe, C., 2011. Towards passive brain–computer interfaces: applying brain–computer interface technology to human–machine systems in general. Journal of neural engineering, 8(2), p.025005."},{"html":"Kopito, R., Haruvi, A., Brande-Eilat, N., Kalev, S., Kay, E. and Furman, D. 2021. Brain-based Authentication: Towards A Scalable, Commercial Grade Solution Using Noninvasive Brain Signals. bioRxiv, 2021.04. 09.439244"},{"html":"Haruvi, A., Kopito, R., Brande-Eilat, N., Kalev, S., Kay, E. and Furman, D. 2022. Measuring and modeling the effect of audio on human focus in everyday environments using brain-computer interface technology. Frontiers in Computational Neuroscience 15, 760561"},{"html":"Furman, D., Reichart, R. and Pratt, H. 2016. Finger flexion imagery: EEG classification through physiologically-inspired feature extraction and hierarchical voting 4th International Winter Conference on Brain-Computer Interface (BCI), 1-4"},{"html":"Furman, D., Benisty, H., Abramovich, T., Ivry, A., Pratt, H. 2016. Enhancement of BCI classifiers through domain adaptation. IEEE International Conference on the Science of Electrical Engineering."},{"html":"Furman, D. Computers Will Soon Read Your Mind: Technology will help patients suffering from ALS or strokes.2023. The Wall Street Journal."},{"html":"Furman, D., Kwalwasser, E., 2023. Interactive electronic content delivery in coordination with rapid decoding of brain activity."},{"html":"Furman, D., Kwalwasser, E., 2021. Empathic Computing System and Methods for Improved Human Interactions With Digital Content Experiences."},{"html":"Lotte, F., Larrue, F., M̈uhl, C. 2013. Flaws in current human training protocols for spontaneous Brain-Computer Interfaces: lessons learned from instructional design. Frontiers in Human Neurosciences, Frontiers, 7 (568)"},{"html":"Bavelier, D., Savulescu, J., Fried, L., Friedmann, T, Lathan, C., Schürle, S., Beard, J,. Rethinking Human Enhancement as Collective Welfarism. Nat Hum Behav. 2019 Mar; 3(3): 204–206. Published online 2019 Feb 11. doi: 10.1038/s41562-019-0545-2"},{"html":"Yerkes RM, Dodson JD (1908). \"The relation of strength of stimulus to rapidity of habit-formation\". Journal of Comparative Neurology and Psychology. 18 (5): 459–482. doi:10.1002/cne.920180503."},{"html":"Taplin, J. 2023. The End of Reality: How Four Billionaires are Selling a Fantasy Future of the Metaverse, Mars, and Crypto. Publisher: PublicAffairs. ISBN: 9781541703155"}]} /> --- # Computers Will Soon Read Your Mind URL: https://arctop.com/deep-dives/computers-will-soon-read-your-mind Published: 2023-12-14

Originally published in The Wall Street Journal on December 14, 2023.

It's been almost a century since psychiatrist Hans Berger made the first electroencephalogram, providing a glimpse into the electric nature of the human brain. EEG readings have helped countless people struggling to recover from ailments ranging from epilepsy and sleep disorders to head injuries and brain tumors. Technology has come a long way since then, and artificial intelligence may soon give us a new brain technology revolution, with advances in the treatment of ALS, strokes and other conditions. As a teenager in a mentorship program, I decided to study the brain after watching a neurosurgeon implant an electrode deep into the brain of a patient with Parkinson's whose tremors were making it impossible for her to hold a pen or drink from a cup. The surgeon implanted the electrode — designed to deliver the right amount of electricity to the exact part of the brain responsible for the tremors — and awoke the patient, her skull still open, to adjust the implant's settings. A few turns of a dial and the shaking stopped. Her tremors were cured. While the discovery of EEG signals was revolutionary, they can be noisy and difficult to interpret, requiring expensive equipment and controlled environments. With recent advances in sensor materials, we are approaching the point at which brain signals can be read throughout the day with comfortable and discreet wearable devices, as a Fitbit or Apple Watch measures our heart rates. Advances in computing and AI mean we could interpret these brain signals in real time. The possibilities include thought-to-speech and thought-to-movement assistive technology for ALS or paralysis patients and accelerated, customized recovery protocols for those suffering from strokes, post-traumatic stress disorder and brain trauma. Brain-computer interfaces could also help personalize teaching and training protocols to fit a learner's cognition and memory processes, eliminate the need for usernames and passwords with a seamless "brain ID," and enable you or a mental-health professional to monitor your emotional state throughout the day. I was part of the team that tried to develop a brain-computer interface for the astrophysicist Stephen Hawking, who suffered from ALS. Hawking's Intel-designed eye-tracking and cheek-click method relied on a level of muscular control that couldn't be taken for granted given his condition. He participated in the project, as he put it, "to assist in research, encourage investment in this area, and, most importantly, to offer some future hope to people diagnosed with ALS and other neurodegenerative conditions." He died in 2018. Today implant-based systems are increasingly powerful and noninvasive, and wearables are improving quickly too. Many of us in the field believe we are nearing an inflection point when countless people will see the fruits of decades of research. The stakes are high. Although every new technology carries promises and risks, few are tied so intimately with who we are.

Mr. Furman is a founder and CEO of Arctop, which makes brain-decoding software.

Source: The Wall Street Journal — "Computers Will Soon Read Your Mind"

--- # Five Levels of Explanation (Part I): How Brain-Computer Interfaces Work URL: https://arctop.com/deep-dives/how-bci-works-part-1 Published: 2023-12-14 What do you think of when you read "Brain-Computer Interfaces" in the title? Elon Musk's Neuralink? Or perhaps you imagined interacting in a virtual world like Neo in "The Matrix" or controlling a Na'vi body in "Avatar"? Media, science-fiction novels, and movies have popularized the concept of "brain interface" technology (for inspiration, consider this list of BCI in fiction), but they have also often led to misunderstandings. Our company prefers the term "Cognition Technology." We'll leave that for a later post. The field of brain-computer interfaces (BCI) – though deeply rooted in science fiction stories and fantasy – is now a solidly established technical field which blends hard science and applied engineering, and is rapidly becoming a mainstream consumer technology after decades of research. BCI is jumping out of the lab and into ubiquity, just as this year the technical field celebrates the 50th anniversary of the coining of the term "BCI" by UCLA researcher Jacques Vidal in 1973 (Vidal 1973).

What is BCI?

A brain-computer interface is a system that measures brain activity, decodes patterns with software, and translates those signals into useful outputs such as metrics, commands, or contextual feedback. Modern BCIs pair non-invasive sensors—like EEG headphones or earbuds—with AI models that run on edge or cloud infrastructure. The goal is to turn cognitive state, intent, or biometric signatures into actionable data in real time while keeping the experience comfortable and safe for everyday use. Every BCI pipeline follows the same loop: sense neural signals, process them with algorithms, and deliver feedback to the user or attached device. Improvements in wearable hardware, signal processing, and machine learning are what make Arctop’s cognition metrics and developer tools possible without implants.
In this post, we aim to help the curious understand what BCI is, explore its capabilities, debunk overhyped claims, and understand its potential impact on our society. Consider this as an invitation to join us in contemplating the future of this technology, its boundaries, emerging opportunities, and how you can be a part of it. Finding answers to these questions is not an easy task. Even among the BCI community, there is disagreement on priorities, opportunities, and capabilities. It's also crucial to differentiate between theoretical possibilities and practical limitations. Just because something can work in theory does not mean it will work in practice. It's vital for us, amidst this technological surge, to establish common ground for future conversation that extends beyond research labs and pop culture, to addressing humanity's broader mission to better itself and possibly co-adapt with artificial intelligence. In this spirit, akin to WIRED's style, we'll explain brain-computer interfaces in five levels of difficulty. We hope you find reading it both fun and enlightening! Let's dive into levels 1 through 3.

Level 1: Child

Imagine telling a computer what to do just by thinking! That's what BCI is about. You don't have to use your hands to type, press any buttons, or even talk. Just put on a cool gadget on your head, like a headband or headphones or special earbuds and these BCI gadgets can "hear" what your brain is thinking and follow your commands. What you can do with this "magic" is only limited by your imagination!

Level 2: Teen

BCIs offer a revolutionary way to interact with computers by translating your brain's impulses into commands. You think about an action, and it happens. All you need to do is to wear a special sensor. BCI sensors can even pick up on your emotions, allowing different apps to personalize your experience in amazing ways. Want to search the Internet, send a message, or play your favorite song? Simply think about it. While this might sound like something out of a sci-fi movie, the past decade has seen incredible strides forward in this field. A blend of advanced scientific knowledge, engineering capabilities, and software development has turned what once seemed like a distant dream into reality. Let's try to understand BCI with a fun analogy. Imagine a football stadium filled with people. Now, picture your brain as this stadium. A bit weird, but you got it, right? If not, the AI-generated image below might help.
In this scenario, each person in the stands is like a single brain cell, called a neuron. As they watch the game, they cheer, shout, chant, holler and sing based on what's happening on the field. BCI sensors capture your brain's activity similar to how microphones placed outside the stadium would pick up the crowd's reactions. Of course, from these "outside" measurements, there are limits. You can't hear what everyone is saying, but you can definitely get the general vibe of the game and what the score is. A loud roar might mean a goal; sudden gasps could indicate a surprising play. You might also pick up from the recordings a "wave" ripple through the crowd or the building excitement during a tense play that ends with one team's fans singing their anthem. Similarly, while a BCI can't decipher every neuron's activity or every single thought in your brain, it can detect changes and patterns in your brain at a level that reflects your thoughts and moods - like whether you're focusing, feeling sleepy, or thinking about moving your hand. Think of it like this: the microphones outside can capture big moments in the game just like BCI sensors can pick up the broader patterns in your brain. You might be wondering, 'if BCI can do so many things, why aren't my friends using it yet?' The best answer we have is that, like the Apple Vision Pro, it's right around the corner. Thanks to advancements in scientific knowledge, better sensors, and more powerful computers with AI, new BCI products are starting to emerge. For now, and for a variety of reasons, BCIs are currently in use by relatively few people: mostly developers, researchers or individuals with disabilities. For example, they help people who've lost the ability to speak or move due to a stroke or certain diseases.

Level 3: College Student

"The most profound technologies are those that disappear. They weave themselves into the fabric of everyday life until they are indistinguishable from it."
Mark Weiser, 1991. BCI, originally developed for medical use, is rapidly evolving into a significant technology for everyday life. A typical BCI system includes three components: the hardware that measures brain activity, the software that interprets this data, and the mechanism that enables interaction with external device(s) and provides user feedback (proposed by Wolpaw et al 2002, one of the foundational papers that pioneered BCI). You can liken these components to the processes of perception, comprehension, and action – what a computer or AI agent needs to interact with you.
There are various brain-sensing modalities and corresponding hardwares to measure brain activity. These include electroencephalography (EEG), electrocorticography (ECoG), and local field potential (LFP) for electrical activity (see the image below); magnetoencephalography (MEG) for magnetic activity; and functional magnetic resonance imaging (fMRI) and functional near-infrared spectroscopy (fNIRS) for blood oxygenation (see Table 1 in Saha et el 2021 for a summary and comparison across these modalities). Among these, EEG – which records the brain's electrical activity by placing sensors on your scalp – is most prevalent in BCIs due to its safety, comfort, fast tracking of the dynamic brain, and affordability.
Revisiting our football stadium analogy, consider BCIs from EEG sensors outside the head to be similar to microphones placed outside the stadium. One way to glean more information about what is happening inside is you can increase the number of microphones (EEG sensors) around various parts of the stadium (the brain). This strategy, "increasing spatial sampling," enhances the detail and robustness of brain activity data, and correspondingly enhances the decoding capabilities of the BCI system. With more comprehensive data, BCI developers can employ sophisticated brain signal processing and machine learning techniques to decode and understand more intricate patterns of brain activity. Now let's delve into what BCIs can actually decipher from brain activity measured via EEG. This technology, akin to an EKG for the brain, has been around for nearly 100 years. EEG continuously captures the brain's activity in tiny voltage fluctuations on the skin, reflecting various experiences – from sensory reactions and movement intention to cognitive processes like attention, memory, and decision making. For instance, when you look at a flashing light, your brain's visual area (occipital lobe) produces oscillatory activities (steady-state visual evoked potentials, or SSVEPs) that reflect the light's flashing frequency. This phenomenon is often described as the brain "entraining" to the stimulus. Similarly, rhythmic sounds produce corresponding activities (auditory evoked potentials, AEPs) in the auditory area (temporal lobe). These responses not only mirror the beats but also the melodies and emotional content in the music. Even imagining movements, like thinking about moving your left arm, induce identifiable changes (event-related desynchronization, ERD) in the motor planning area (parietal lobe). These diverse signals, all detectable by EEG, have been instrumental in advancing BCI applications over the last two decades.
EEG can also reveal different brain states. For instance, closing your eyes induces an instant and distinct power increase in oscillations at around 10 cycles per second in your visual cortex. Sleep stages, from light to deep sleep, are marked by increasingly synchronized, broad, and slow-oscillating brain activities (known as slow waves, oscillating at just a few cycles per second). Interestingly, the EEG patterns during dreaming resemble those of wakefulness. Recent studies have further expanded EEG's scope, demonstrating its ability to capture various mental states, from changes in attention and emotional states to varying levels of drowsiness. The brain activities captured by EEG are subtle, often only a few microvolts in amplitude. They're easily overshadowed by non-brain activities like muscle or eye movement, which can be ten or hundred times stronger. This complexity demands that BCI developers have a deep understanding of signal processing and machine learning to accurately interpret the faint EEG data, especially in real-time, real-world scenarios.
Over the past two decades, we have started to see more developments and applications for mainstream uses, from enhancing computer games and virtual-reality experiences, providing feedback to promote meditation training, tracking health metrics like sleep, and integrating into learning with content adapted to student's mental states. In our next post, we'll go into details at a graduate level into how BCIs work and are increasingly advancing into the fabric of life. Levels 4 and 5 coming soon…

Further reading: Reinforcement Learning from Brain Feedback (RLbF) for Large Language Model Improvement — our paper on using decoded EEG as a training signal for large language models (preprint: 10.20944/preprints202608.1274.v1).

Brain-Computer Interface FAQ

What is a brain-computer interface?
A brain-computer interface (BCI) is a system that measures brain activity, decodes it with software, and turns those signals into outputs such as metrics, commands, or feedback for a connected device.
Do BCIs require surgery or implants?
Most consumer and enterprise BCIs today are non-invasive. They use wearable sensors like EEG headsets or earbuds to capture brain activity through the scalp, so no surgery is required.
What can BCIs do today?
Modern BCIs can monitor cognition metrics such as focus or engagement, enable adaptive software experiences, support accessibility use cases, and provide biometric authentication based on unique brain signal signatures.
--- # Brain-based Authentication: Towards A Scalable, Commercial Grade Solution Using Noninvasive Signals URL: https://arctop.com/deep-dives/brain-based-authentication Published: 2021-04-09
This paper is available in PDF format here.

Abstract

Here we report on a field test where we asked if it is feasible to deliver a scalable, commercial-grade solution for brain-based authentication given currently available head wearables. In this study, forty-nine (49) participants completed multiple sessions in their natural home environment over a single week. Participants used an off-the-shelf brain signal measuring headband to record their own brain activity while completing various tasks. Recording sessions were self-operated by the participants and unsupervised by any expert or technician to simulate real world use cases, while also contrasting common research approaches to this topic that rely on data from controlled laboratory conditions. Although brain signals have a non-stationary, complex nature, when participants watched rapidly presented images, our authentication system was able to successfully construct a unique and robust "brain ID" for each participant. Based on this brain ID, we developed a simplified brain-based authentication method that captures distinguishable information with reliable, commercial-grade performance from participants at their own homes. We conclude that noninvasively measured brain signals are ideal for use in biometric authentication systems, especially in environments where head wearables such as headphones or AR/VR devices are used as these devices offer a natural form factor for capturing participant brain ID continuously.

Introduction

When a user requests access to a system or a device, an authentication process must confirm whether the identity claim of the user is genuine or whether they are an imposter. A simple example of an authentication method is an alphanumeric password like 'abc123,' while a more complex example is a digital fingerprint captured by a smartphone sensor. Effective authentication is critical to security for both consumers and enterprises. Because of the high frequency of use of authentication systems, methods need to be both convenient and secure. That balance - between convenience and security - is a defining performance characteristic of authentication systems. The strongest authentication systems are very secure and, often, very cumbersome to implement and maintain. In contrast, weak authentication methods are very convenient, but have been responsible for countless data breaches because of their equivalent ease of being hacked. The "password chaos" of modern life seems to have reached a boiling point and it is clear that future computing systems need improved methods that both deliver increased security along with an increased convenience that ensures adherence, at the human level of behavior, to security protocols. Biometric authentication is any method that uses natural occurring information to verify a user's identity. Many biometric authentication systems have already been developed based on fingerprints, faces, palm veins, irises, voices, gaits, and other metrics. There are significant advantages to using biometrics for authentication, since user experience is convenient and fast, it is non-transferable, and usually the system can reach very high performances (high false rejection rate, and low false acceptance rate) typically without requiring much attention, if any, from the user. There are disadvantages as well however, since biometrics can be faked or stolen and when that happens, the victim cannot simply replace them to avoid impersonation. Certain biometrics, for example fingerprints or faces, can be easily captured today by cameras remotely without any knowledge from the individual that is being surveilled, and once a face or finger ID is compromised, the remedy is extremely difficult. Brain-based authentication is the process of verifying an individual's identity by using their brain signal, and as an approach it offers several distinct advantages over other biometric authentication methods. Since at least the late 1980s1 neuroscientists have observed that noninvasively measured human brain signals carry personally identifying information2,3 that differentiates between family members and across a broad population.4,5 Brain signals, unlike many other biomarkers are concealed: an invisible signal that is never exposed in daily life. Second, brain signals are dynamic, non-stationary and extremely complex. They are the result of a unique series of brain waves superpositioning in a given brain at any moment, and these waves reflect both personal brain function and anatomy. Taken together, this makes brain signals an ideal candidate for use as a biometric6,7,8 method. Indeed, many groups have attempted to build biometric authentication systems based on brain signals.9,10,11,12,13,14,15,16,17,18,19 Generally, the process involves a machine-learning classifier to identify if a given brain signal belongs to a genuine identity or to an imposter one (Figure 1).
Figure 1: Schematic illustration of the brain-based authentication process. Brain signal is recorded while participants watch images (\"Prompt\"). The brain response (e.g. \"John's brain signal\") is fed into a trained classifier of that participant. The classifier decides if the brain pattern matches the participant (authenticates \"John\") or not (rejects)."} /> The overall usability of brain-based authentication systems has been increasing since 2010,20 however most are still far from proving field-viability and bringing new value and utility to existing authentication providers.21,22 Brain data is commonly collected in a laboratory under controlled conditions in other studies, where a trained technician is an essential part of the brain measurement procedure. Often authentication performance tests are also done with a small number of people, with all measurements taking place in a single session despite the common industry knowledge that single session data maps poorly to daily consumer electronics usage, where individuals put devices on and off regularly.23,24,25,26 While brain biometric identity appears to be one of the most natural, powerful methods for head wearables, its robustness has not been sufficiently vetted in real world conditions that parallel the end use cases such as:
  • Professionals who work remotely and wear headsets as part of their daily job, which requires them to be authenticated across different applications throughout the day.
  • Gamers who want a seamless, hands-free and voice-free method of profile loading and authorizing in-game purchases.
  • e-Commerce consumers whose check-out experience is currently interrupted by passwords.
  • Surgeons, heavy machinery operators, medical professionals and others working in high-strain, sanitary environments that require especially high reliability and convenience.
  • Air gapped environments where there are strict demands on performance, confidentiality and all biometric processing and decisions need to be performed on-edge devices.
To model these use-cases, we set out to perform a generalizable field test of brain-based authentication using brain signals measured noninvasively from people in their natural home environment. In other words, their real world context. This feasibility test "in the wild" advances the applied science of brain biometric analysis towards scalable implementations as all participants were completely new (naïve users) to the system and enrolled themselves from home. They used a comfortable head wearable for hours during test sessions, and this device had minimal data and battery requirements. They performed repeated authentication attempts across several different days, and individual brain ID's were shown to be robust against changes in brain state and ambient noise inherent to brain data.

Methods

Participants

Sixty-two (62) participants were recruited to complete four (4) sessions over a single (1) week at their own home. Adult participants were recruited from an opt-in screening panel and came from all five (5) major regions of the continental United States (Northeast, Southwest, West, Southeast, and Midwest). Only participants who reported normal vision, or vision that was corrected to normal with contact lenses were included. We excluded volunteers who reported using medication that might influence the experiment or other neurological or psychiatric conditions that could influence the results. Written informed consent was obtained from all participants before screening and the main experimental sessions. Thirteen (13) participants were ultimately excluded for problematic survey response patterns within the study and/or invalid brain data, leaving 49 participants (mean age= 36, SD=8.25, 16 females) enrolled and eligible to be included in the analysis.

Sessions

Individuals participated in the study by recording sessions from their own homes at their own pace, over one week as detailed in Haruvi et al 2021.27 Each participant received an Arctop technology kit that included headphones (Sony), a brain signal measuring headband (InteraXon) and a tablet computer (Samsung) with a designated app (Arctop). Each participant recorded four sessions, one hour long each where towards the end of each session, six (6) authentication events were presented. Each authentication event started with a message declaring the upcoming event and instructing the participant to stay steady. Then, a fixation period which enabled the participant to get prepared (2 seconds) before seeing the rapid serial visual presentation (RSVP) of selected images at 10Hz for 10 seconds (Figure 2).
Figure 2: Time course of brain-based authentication using rapid serial visual presentation prompts. At each authentication event a sequence of images rapidly (10Hz) changes for 10 seconds while the brain response is recorded. In each session, six RSVP events were presented to each participant."} />

Data Acquisition

Participants engaged in a variety of tasks during each session while their electrical brain activity was recorded using InteraXon's Muse-S device, a portable, noninvasive electroencephalography (EEG) device weighing 41 grams (Figure 3, left panel). The device includes four dry fabric EEG sensors (sampling rate: 256 Hz), photoplethysmography (PPG) sensors (for heart rate) and motion sensors (gyroscope and accelerometer). The EEG sensors are located on the scalp, two frontal channels (AF7 and AF8) and two temporals which rest behind the ears (TP9 and TP10), with a reference channel at Fpz. The headbands were put on by the participants themselves, with the assistance of a Quality Assurance (QA) screen that started before each session. The QA showed the participants, in real-time, the channels' quality, easily directing them to adjust the headband properly for optimal signal quality.
Figure 3: Schematic illustration of the processing pipeline. Data acquisition included at home brain data recordings of 4 sessions, each on a different day. Each session included 6 RSVP videos (Figure 2). Brain data processing included filtering the signal, feature extraction and training a machine learning authentication classifier per participant. The classifier decides if the input belongs to the participant (Genuine) or not (Imposter)."} />

Preprocessing and feature extraction

Data analysis was performed only for periods within the RSVP events. A band-pass filter (0.5-46Hz) was applied on each channel. The filtered signal of RSVP event was segmented into 31 epochs of 2 seconds in length, using a sliding window with a stride of 250ms (Figure 3, middle panels). Comprehensive feature extraction and engineering was not the goal of this current study. Here we aimed for effective information capture without deeper optimization to first test the core principles. Accordingly, for each epoch and for each EEG channel, the following features were calculated: The average power for each of the traditional frequency bands (Alpha, Beta, Gamma, Delta, Theta), power spectrum interactions (engagement index, Alpha over Delta, Beta over Theta, Theta/Alpha), time domain features such as averages, standard-deviations, kurtosis, entropy and number of zero-crossing points, and pairwise correlations between channels for the various frequency bands. All together, for each epoch, a total of 140 features were extracted.

Models training and testing

For each participant we had a total of 24 RSVP events (Supp. Video 1), which we collected over the 4 sessions. For each participant, three sessions (18 events) were chosen randomly to be used for training (Figure 3, right panels). The fourth session was used for testing (6 events). Authentication prediction of an event acts in two steps, the first at the epoch level, where each epoch is determined to belong to a genuine or imposter. Second is the final decision regarding the whole event identity (genuine or imposter). For each participant an authentication model was trained first at the epoch level. Model classification was done with XGBoost classifier (binary classification). The labeling of the data was changed in accordance to the identity under training. Epochs (feature space, 140 features per epoch) of genuine identity were labeled as one (558 epochs), while epochs from the rest of the participants were of imposter identity, and labeled as zero (26784 epochs). Thirty percent of training data (random and balance split) was dedicated for validation and to determine epoch thresholding. Standardization procedure over the training epochs was applied. Later, standardization means and stds (standard deviation values) of the training features were used to normalize the validation and testing data. Epoch's threshold for classification was optimized to minimize false acceptance rate (FAR), while maximizing true rejection rate (see Supp. Figure 3). Identity predictions of validation data epochs, after thresholding, exhibited high accuracy for all participants (average accuracy=0.9865, STD=0.00929). A final decision about participant identity was given at the event level. Event threshold, as before, was determined by an optimization algorithm, but here it was done over the validation data. For the validation data, after thresholding, the average event authentication accuracy over all participants was 0.9965, with STD=0.00041. Since per each participant validation data included 264 events, it suggests that on the average, after thresholding all events were identified correctly except one. Test data included 294 events and 9114 epochs. Training model predicted the identity probability of each epoch. Probabilities above the epoch threshold were determined to be of a genuine identity, while those below the threshold are of imposter identity (Supp. Figure 4A-C). Test event was declared to be of genuine identity only if 40% of its epochs were above the epoch threshold (Supp. Figure 4A1-C1).

Results

In our authentication system we derive from non-invasively recorded brain signals a "brain ID" abstraction that proved to be representative of each participant, and differentiating from one another. The brain response during a RSVP event is used as a brain biometric ID for identity verification. In order to demonstrate characteristics of this brain ID, we will follow the example presented in Figure 4. In our system, the authentication period depends on the event length, here it is approximately 10 seconds (one RSVP event). The event is composed of 4 channels (256Hz), segmented into 31 epochs, 2 seconds long, with a stride of 250ms (Figure 4A-B). The non-stationary nature of the brain signal, and the fact that it is a superposition of hundreds of simultaneous processes in the brain, makes the signal unique in time, unrepeatable, and unpredictable. Even when a user's brain is stimulated by identical stimuli, no two epochs are alike (Figure 4B), nor are two events alike. In Figure 4C and in Figure 4D the epochs of two events (the brain signal space) of the same participant are presented respectively. For each event, the epochs are aligned vertically, ordered in time, creating a visualization of the brain ID. This representation enables us to see easily that none of the epochs are identical, nor are the full events.
Figure 4: Event epoching.
A. The RSVP authentication event is a normalized filtered brain signal response composed of four channels of EEG data, 10 seconds long. Each event signal is segmented into 31 epochs (where each epoch carries 4 channels), 2 seconds in length, and with a sliding window of 0.25 sec stride. B. Channel-1 of epochs #5, and #28 (top, bottom) are shown for demonstration. Note, that the shaded areas colored in green and red in panel A correspond for these epochs respectively. C. The epochs of the event signal in A, can be rearranged into an image (here again just channel-1 is shown). Where each row is an epoch, and the epochs are time ordered vertically. In C and D, events which were taken from the same participant (#39), but from different sessions are shown. E, F the corresponding features of the epochs presented in C, D are presented. Note that the calculation of epoch features involves all epoch channels. While the non-stationary nature of the EEG data dictates that the events (as shown in C, D) do not resemble each other, the features images (E, F) demonstrate high similarity."} /> In contrast, the brain ID data becomes highly correlated when the same events shown previously in Figure 4C-D are now presented at a higher level of analysis (the features space) (Figure 4E-F). High correlation is visible among epochs of the same event, creating a clear brain ID pattern. As one can note, a similar pattern is carried by brain data captured at different occasions, and we can conclude that usage of an event instead of a single epoch for deriving the brain ID increases the pattern robustness and increases both the sensitivity (true acceptance rate) and specificity (true rejection rate) of the system. In Figure 5A-D, four brain IDs examples of different participants are presented. It is apparent that each brain ID carries a unique pattern that is distinguishable from the others. We would like to generalize the idea of using RSVP events brain IDs as a verification method in our authentication system over all the participants. If the event brain ID is used as an identity verification two criteria must be fulfilled:
  1. The similarity between different events brain IDs of the same person is kept high: even and especially, for events which were recorded at different occasions.
  2. The brain ID of each participant is unique, and distinguishable.
Figure 5: Similarity among intra and across inter participants events. Panels A, B, C, and D show the features of a single event for different participants (#23, #35, #,39, and #34 respectively). The pattern of an event appears more robust, as the features values are repeatedly conserved across many epochs. On the other hand, it looks like for each participant the pattern is specific. The similarity (or dissimilarity) between events can be measured by a correlation coefficient. In E we present the event correlation matrix, where element Eij, is the average pairwise correlation across all training events of participant i and participant j. Note that the intra-correlation coefficients (diagonal) are usually higher than inter-correlation (off-diagonal), suggesting that for the same participant the pattern of different events is conserved, and patterns of different participants are different. This understanding leads us to the idea of an authentication system by events. Also note that the order of the participants in E, is in accordance with the hierarchy cluster tree shown in Supp. Figure 1. The matrix here is normalized."} /> The similarity between two events (at the feature space) can be measured by the Pearson correlation coefficient between the means of the events. Thus the similarity between two participants is the mean of all pairwise events correlations of these participants. In Figure 5E, the normalized correlation matrix across all participants is presented. Values are represented by colors (colorbar 0.2-1), higher values suggest higher similarity. The order of participants along the axes was determined using a hierarchical clustering algorithm (see Supp. Figure 1). The averaged similarity between events belonging to the same participant (intra correlation) are along the diagonal elements of the correlation matrix, while the averaged similarity between events of two different participants (inter correlation), are the off diagonal matrix. In general, we have found that for all participants, the similarity of intra correlation is higher than the inter correlation (Figure 5E, Supp. Figure 2). Looking more deeply, histograms in Supp. Figure 2, shows that most of the inter- and intra-participant correlation are indeed separated: for more than half of the participants the intra-correlation is higher than 0.7, where most of the inter-correlations are lower than 0.35. The inset in Supp. Figure 2 also suggests a linear relation between the mean inter-correlation of a participant and its intra correlation. Namely, participants having relatively low intra-correlation (~0.5), their inter-correlations will be low as well (~0.25). These results reflect that criteria 1 and 2 (above) are fulfilled, and the brain IDs can be used for identity classification. As described in the Method section, for each participant an authentication model and relevant thresholds were found. These models were tested on the test data which in total included 249 genuine events, and 14112 imposter events. The general performance of our authentication system is summarized in Figure 6. The averaged false acceptance rate (FAR) is 9% and the false rejection rate (FRR) is 13%, making the solution sufficient for certain commercial authentication use-cases, but not all. The averages shown here are the means over the individuals' FAR, FRR.
Figure 6: Summary of authentication performance in the field. Forty nine participants were included in the test. Each participant had six genuine events, and 288 imposter events. All together in this test we used 294 genuine events, and 14112 imposter events. In A, and B histograms of authentication system performance at the participant level is present. In A, the performance regarding imposters (true rejection rate, and false acceptance rate). In B, the performance regarding genuines identities (true acceptance rate, and false rejection rate). These values were first calculated per each participant, and then distribution was calculated. C. A confusion matrix summary, showing the averaged performance over all participants. A detailed performance summary per each participant can be found in supp. Table 1."} /> Out of the total number of participants in this experiment (49), 37 participants have FRR=0, where 24 participants have FRR=0 and FAR<=9% (Supp. Table 1). Next we asked what will be the performance of the authentication system when only certain brain signal information is considered. Explicitly, we repeated the training procedure (Methods) but this time with only the power spectrum features of the following brainwave modes: Delta(0.5-4Hz), Theta(4-8Hz), Alpha(8-12.5), Beta(12.5-30Hz) and Gamma(30-48), and with some combinations (Alpha-Beta, and Theta-Alpha-Beta). We found that usually for these features, FRR can reach low values while the FAR is always kept high (Table 1). As the number of features is increasing, the better the performances of the authentication system. This implies that more sophisticated models such as deep neural networks will greatly improve the performance of the authentication system. We will report on the results of different systems such as these in future field test reports.
Table 1: Model performance as function of feature types. The same training and testing datasets were used for all models. The power spectrum density (PSD) of the following frequency bands were used as features. Delta(0.5-4Hz), Theta(4-8Hz), Alpha(8-12.5), Beta(12.5-30Hz) and Gamma(30-48). Each bandpass has four features, corresponding to the number of brain data channels. When using only one type of powerband feature, the averaged FRR may reach low levels, but the FAR always remains high."} />

Discussion

We performed a generalizable field test of a brain-based authentication system that uses non-invasively measured brain signals to verify user identity. All participants were completely new (naïve users) to the system, enrolled themselves from home in a self-guided tutorial, used a comfortable head wearable for hours at a time without issue and performed repeated authentication measures across multiple days. On the whole, this amounts to a reasonable simulation of real contexts that enterprise and consumer authentication methods need to operate in to be commercially viable. Specifically, these methods must work regardless of time of day and be robust to changes in brain state (pre/post-coffee, hunger, wakefulness, awareness, etc.) and the ambient noise inherent to measurements made outside of controlled laboratory conditions. The main goal of this field test was to evaluate the base feasibility of a scalable, commercial-grade brain ID authentication system; advanced data engineering methods were not applied to boost performance further. A simplified feature set and simple machine learning methods were applied over a minimal training period of less than three minutes enrollment data per participant. Amidst these severe constraints on performance, brain-based authentication proved to be approaching commercial-grade levels. In future tests the parameters used will be optimized, here our authentication system ran on suboptimal parameters that were fast calculated to serve as more heuristics than anything. For example the epoch length, the authentication event duration, selection of features by their importance, or the amount of training data we know have an impact on performance from previous and ongoing work. These are all tunable parameters depending on the demands of the authentication task: future research will clarify the timescales at which the optimal information for identification verification occurs for each tier of authentication system. We are confident that more sophisticated machine-learning architectures together with other parameters optimization will deliver superior, product level authentication performance that will match or exceed the performance of top non-brain authentication biometrics available today. Wearables that touch the head, such as headphones or AR/VR, are a natural form factor for brain-based authentication and we anticipate that demand from enterprises and consumers will necessitate that these devices evolve beyond passwords and fingerprints to iris ID based on eye scanning and eventually brain ID, based on brain scanning. The demand for both strong and convenient authentication solutions for future head-worn devices drove our design of the paradigm for prompt-response analysis here, and it is notable that the rapid image prompt-response paradigm evaluated (with users watching images on a tablet while wearing a headband) has been validated by us elsewhere in AR (Supp. Video 2) using Microsoft Hololens. Given the performance obtained in this field test and the ease-of-use of this method for head wearables, brain ID seems to be one of the most intuitive and powerful authentication solutions for next generation head-worn computers. Brain identities, like any other biometric identity, will need to conform to privacy standards and be offered within protected software and chip architectures such as those pioneered for fingerprint scanners and face recognition, but this is no limitation on the adoption of such a beneficial method. Biometrics as a class are uniquely comfortable and convenient to use because they do not require the user to remember anything (like a password), or carry anything (like a physical key). Biometrics offset the cognitive load of password management plus the risks associated with alphanumeric passwords, and even offer the promise of obviating passwords altogether in future computing ecosystems. For now, brain ID is at a nascent stage of industry adoption and the solution presented here represents one of the more scalable designs, since we can easily increase the dataset to more participants and more events within the principled framework of forcing divergences in inter-participant data and convergences in intra-participant data. Furthermore, the head wearable that people put on themselves in this test to measure their brain signal is a consumer device that is currently available worldwide, highlighting the lack of need for exotic or rare materials to acquire sufficient brain signal to measure brain IDs, nor the need for specialized laboratories or facilities. More information in the brain signal remains unexplored here, being outside the scope of the current field test and report. Future research will develop concepts related to the theoretical and practical information boundaries in brain signal, since for head wearables in particular, brain biometric identity warrants continued testing across expanded participant populations and implementation in commercial devices that are optimized for given use-cases and environments.

Conclusion

We showed that a relatively simple brain-based authentication system can use non-invasively measured brain signals from consumer quality head wearable devices to differentiate between users with a high degree of certainty. Authentication using non-invasively-measured brain signals in this way was found to not only be feasible, but robust: the correlation matrices derived from the current test find our computed brain identities to be readily distinguishable between different participants and consistently similar among participants, satisfying the core requirements of a commercial-grade biometric authentication system. The complexity inherent to human brain signals was, therefore, found to not be too volatile to be leveraged for steady, reliable use as a passwordless authentication method. We built and validated the method through a scalable software infrastructure that was able to deliver brain-based authentication at a commercial-grade, within a generalized framework that provides for continual performance improvement with additions of new participants. As the methods were designed around characteristic patterns observable during limited windows of time, at any time, it is clear that there is value to continued data collection at larger scales and across additional contexts. For both inter-subject variability and to further clarify the invariant patterns underlying intra-participant variability, expanded data collection can be beneficial. The present sample is sufficient however to conclude that brain-based authentication is already a viable method for certain commercial uses, and has the potential to serve many more in the future.

Ethics declarations

All authors are employees of Arctop Inc.

Supplementary Materials

Supp. Figure 1: Brain biometric ID cluster tree. The mean overall training events were calculated for each participant. Mean event correlation matrix between participants was then calculated by pairwise correlation. Using this matrix, the hierarchical cluster tree (dendrogram) algorithm creates the linkage distance between participants (y-axis)."} />
Supp. Figure 2: Histograms of intra-participant and inter-participants events correlations. Intra-participant events correlation is the mean of pairwise correlation between all training events of a participant with themselves. Inter-participant correlations are the mean of pairwise correlation of all training events of a participant with the events of another participant. The diagonal elements in the event correlation matrix (Figure 6E), represents the intra-participant correlations while the inter-participants correlations are the off-diagonal elements of the matrix. It is clearly seen that intra-participant correlations are generally higher than the inter-participants correlation. Meaning, a higher similarity within intra events patterns compared with inter-participants events. Although there is an overlap between the two histograms, it does not necessarily mean that perfect separability at the authentication event level is not feasible, as suggested by the inset."} />
Supp. Figure 3: Epoch predictions and threshold determination. Three sessions per participant are contributing to the training data. Out of it, 30% are devoted for model validation, and to determine the model threshold. In panels A, B, C the prediction of three models trained for three participants (sub #23, # 35, #34 respectively) are presented for the validation data. Here, epochs predictions of genuine identity are marked in black dots, and epochs predictions of imposters are marked in red. The threshold (black dashed line), discernmenting between genuine and imposter epochs is determined by an optimization algorithm. The algorithm finds a threshold probability in which the false acceptance rate (FAR) is minimal while the true acceptance rate (1-FRR) is maximal. This is under the condition for TAR>90%, and FAR<3%. This is demonstrated in panels A1, B1, C1. FAR, TAR functions are plotted in red and black respectively, the threshold which was found is marked in black dashed-dot line, y-axis is in logarithmic scale."} />
Supp. Figure 4: Event prediction. The events are segmented into 31 epochs. The probability of each epoch to be of a genuine identity or of an imposter one is determined by a model. Probabilities above the threshold (black dash line) belong to genuine identity, and if below the threshold, to an imposter. Threshold was determined previously in the training process (see Supp. Figure 3). A, B, C the predictions of three models trained for three participants (sub #23, # 35, #34 respectively) are presented. Here the epochs under test are only of genuine identity (black dots). While in A, and C all predictions are above threshold, in B some of the predictions are below the threshold. The final decision whether the event is of genuine identity is determined only if more than 40% of epochs are above the threshold. In A1, B1, C1 full test prediction is shown for the same three participants. The test included 294 events, where each participant has 6 genuine events. Events of genuine identity are marked in black dots and imposter events are in red dots. In all three cases all genuine events were identified correctly, having zero false rejection rate (FRR=0). As for the imposters, only in C1, all imposter events are below the threshold, with zero false acceptance rate (FAR=0). The final FRR and FAR of each participant is shown in the yellow windows."} />
Supp. Table 1: A detailed performance of the authentication system for each participant. The coefficients of the confusion matrix per each participant is presented."} /> Supp. Video 1: Rapid Serial Visual Presentation (RSVP) Stimuli. Example of a stream of images watched by participants while brain signals were recorded by their headband. https://youtu.be/TWUzbX3Q8sk Supp. Video 2: Brain-based Authentication: Living Room Demo. Microsoft HoloLens 1, retrofitted with BCI sensors, delivers passwordless authentication. https://youtu.be/n6v9z3lNs2M Electroencephalography and clinical neurophysiology 66, no. 6 (1987): 489–501."},{"html":"Poulos, Marios, Maria Rangoussi, and Nikolaos Alexandris. \"Neural network based person identification using EEG features.\" In 1999 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings. ICASSP99 (Cat. No. 99CH36258), vol. 2, pp. 1117–1120. IEEE, 1999."},{"html":"Poulos, M., M. Rangoussi, N. Alexandris, and A. Evangelou. \"Person identification from the EEG using nonlinear signal classification.\" Methods of information in Medicine 41, no. 1 (2002): 64–75."},{"html":"Marcel, Sebastien, and José del R. Millán. \"Person authentication using brainwaves (EEG) and maximum a posteriori model adaptation.\" IEEE transactions on pattern analysis and machine intelligence 29, no. 4 (2007): 743–752."},{"html":"Van Beijsterveldt, C. E. M., and G. C. M. Van Baal. \"Twin and family studies of the human electroencephalogram: a review and a meta-analysis.\" Biological psychology 61, no. 1-2 (2002): 111–138. "},{"html":"Thorpe, Julie, Paul C. Van Oorschot, and Anil Somayaji. \"Pass-thoughts: authenticating with our minds.\" In Proceedings of the 2005 workshop on New security paradigms, pp. 45–56. 2005."},{"html":"Wang, Min, Jiankun Hu, and Hussein A. Abbass. \"BrainPrint: EEG biometric identification based on analyzing brain connectivity graphs.\" Pattern Recognition 105 (2020): 107381."},{"html":"Piplani, Tanya, Nick Merill, and John Chuang. \"Faking it, Making it: Fooling and Improving Brain-Based Authentication with Generative Adversarial Networks.\" In 2018 IEEE 9th International Conference on Biometrics Theory, Applications and Systems (BTAS), pp. 1–7. IEEE, 2018."},{"html":"Abo-Zahhad, Mohammed, Sabah Mohammed Ahmed, and Sherif Nagib Abbas. \"State-of-the-art methods and future perspectives for personal recognition based on electroencephalogram signals.\" IET Biometrics 4, no. 3 (2015): 179–190."},{"html":"Armstrong, Blair C., Maria V. Ruiz-Blondet, Negin Khalifian, Kenneth J. Kurtz, Zhanpeng Jin, and Sarah Laszlo. \"Brainprint: Assessing the uniqueness, collectability, and permanence of a novel method for ERP biometrics.\" Neurocomputing 166 (2015): 59–67."},{"html":"Ashby, Corey, Amit Bhatia, Francesco Tenore, and Jacob Vogelstein. \"Low-cost electroencephalogram (EEG) based authentication.\" In 2011 5th International IEEE/EMBS Conference on Neural Engineering, pp. 442–445. IEEE, 2011."},{"html":"Campisi, Patrizio, and Daria La Rocca. \"Brain waves for automatic biometric-based user recognition.\" IEEE transactions on information forensics and security 9, no. 5 (2014): 782–800."},{"html":"Maiorana, Emanuele, Daria La Rocca, and Patrizio Campisi. \"On the permanence of EEG signals for biometric recognition.\" IEEE Transactions on Information Forensics and Security 11, no. 1 (2015): 163–175."},{"html":"Mohanchandra, Kusuma, G. M. Lingaraju, Prashanth Kambli, and Vinay Krishnamurthy. \"Using brain waves as new biometric feature for authenticating a computer user in real-time.\" International Journal of Biometrics and Bioinformatics (IJBB) 7, no. 1 (2013): 49."},{"html":"Mu, Zhendong, Jianfeng Hu, and Jianliang Min. \"EEG-based person authentication using a fuzzy entropy-related approach with two electrodes.\" Entropy 18, no. 12 (2016): 432."},{"html":"Palaniappan, Ramaswamy. \"Two-stage biometric authentication method using thought activity brain waves.\" International journal of neural systems 18, no. 01 (2008): 59–66."},{"html":"Paranjape, R. B., J. Mahovsky, L. Benedicenti, and Z. Koles. \"The electroencephalogram as a biometric.\" In Canadian Conference on Electrical and Computer Engineering 2001. Conference Proceedings (Cat. No. 01TH8555), vol. 2, pp. 1363–1366. IEEE, 2001."},{"html":"Ruiz-Blondet, Maria V., Zhanpeng Jin, and Sarah Laszlo. \"CEREBRE: A novel method for very high accuracy event-related potential biometric identification.\" IEEE Transactions on Information Forensics and Security 11, no. 7 (2016): 1618–1629."},{"html":"Thomas, Kavitha P., and A. Prasad Vinod. \"EEG-based biometric authentication using gamma band power during rest state.\" Circuits, Systems, and Signal Processing 37, no. 1 (2018): 277–289."},{"html":"Yang, Su, and Farzin Deravi. \"On the usability of electroencephalographic signals for biometric recognition: A survey.\" IEEE Transactions on Human-Machine Systems 47, no. 6 (2017): 958–969."},{"html":"Revett, Kenneth, and Sergio Tenreiro de Magalhães. \"Cognitive biometrics: Challenges for the future.\" In International Conference on Global Security, Safety, and Sustainability, pp. 79–86. Springer, Berlin, Heidelberg, 2010."},{"html":"Curran, Max T., Jong-kai Yang, Nick Merrill, and John Chuang. \"Passthoughts authentication with low cost EarEEG.\" In 2016 38th Annual international conference of the IEEE engineering in medicine and biology society (EMBC), pp. 1979–1982. IEEE, 2016"},{"html":"Del Pozo-Banos, Marcos, Jesús B. Alonso, Jaime R. Ticay-Rivas, and Carlos M. Travieso. \"Electroencephalogram subject identification: A review.\" Expert Systems with Applications 41, no. 15 (2014): 6537–6554."},{"html":"Marcel, Sebastien, and José del R. Millán. \"Person authentication using brainwaves (EEG) and maximum a posteriori model adaptation.\" IEEE transactions on pattern analysis and machine intelligence 29, no. 4 (2007): 743–752."},{"html":"Näpflin, Markus, Marc Wildi, and Johannes Sarnthein. \"Test–retest reliability of resting EEG spectra validates a statistical signature of persons.\" Clinical Neurophysiology 118, no. 11 (2007): 2519–2524."},{"html":"Näpflin, Markus, Marc Wildi, and Johannes Sarnthein. \"Test–retest reliability of EEG spectra during a working memory task.\" Neuroimage 43, no. 4 (2008): 687–693."},{"html":"Haruvi Aia, Ronen Kopito, Noa Brande-Eilat, Shai Kalev, Eitan Kay, and Dan Furman. \"Differences in the effects on human focus of music playlists and personalized soundscapes, as measured by brain signals.\" Biorxiv (2021)."}]} />
--- # Modeling The Effect of Background Sounds on Human Focus Using Brain Decoding Technology URL: https://arctop.com/deep-dives/modeling-focus Published: 2021-04-02 This paper is available in PDF format here.

Abstract

The goal of this study was to investigate the effect of sounds on human focus and to identify the properties that contribute most to increasing and decreasing focus in people within their natural, everyday environment. Participants (N=62, 18-65y) performed various tasks on a tablet computer while listening to either no background sounds (silence), popular music playlists designed to increase focus (pre-recorded songs in a particular sequence), or engineered soundscapes that were personalized to individual listeners (digital audio composed in real-time based on input parameters such as heart rate, time of day, location, etc.). Sounds were delivered to participants through headphones while simultaneously their brain signals were recorded by a portable electroencephalography headband. Participants completed four one-hour long sessions at home during which different sound content played continuously. Using brain decoding technology, we obtained individual participant focus levels over time and used this data to analyze the effects of various properties of sound. We found that while participants were working, personalized soundscapes increased their focus significantly above silence (p=0.008), while music playlists did not have a significant effect. For the young adult demographic (18-36y), all sound content tested was significantly better than silence at producing focus (p=0.001-0.009). Personalized soundscapes increased focus the most relative to silence, but playlists of pre-recorded songs also increased focus significantly during specific time intervals. Ultimately we found that it is possible to accurately predict human focus levels that will be experienced in response to sounds a priori based on the sound's physical properties. We then applied this finding to compare between music genres and revealed that classical music, engineered soundscapes, and natural sounds were the best genres for increasing focus, while pop and hip-hop were the worst. These insights can enable human and artificial intelligence composers to produce increases or decreases in listener focus with high temporal (millisecond) precision. Future research will include real-time adaptation of sound libraries for other functional objectives beyond affecting focus, such as affecting listener enjoyment, stress, and memory.

Introduction

The effect of sound on human experience

Sounds are all around us, from natural sounds like the wind, to engineered sounds like music. It is well-established that sounds have a major influence on the human brain and consequently, human experience (Levitin, 2006; Sacks, 2010). Sounds can reduce stress (Davis & Thaut, 1989), support learning and memory formation (Hallam et al., 2002), improve mood (Chanda & Levitin, 2013), and increase motivation (Salimpoor et al., 2015). Sounds can also do the opposite and create aversive experiences (Kumar et al., 2012; Schreiber & Kahneman, 2000; Zald & Pardo, 2002). One of the most significant effects of sounds is to impact focus. Focus is commonly demanded by tasks of daily living and work, and in these areas sounds offer a safe way to increase focus levels and productivity. However, sounds can be beneficial or distracting, and previous results have been inconclusive in determining the reason (de la Mora Velasco & Hirumi, 2020). For example, it has been found that listening to music with lyrics while reading or working can decrease concentration or cognitive performance (H. Liu et al., 2021; Shih et al., 2012), while several studies have shown oppositely that natural-occurring sounds such as white noise, as well as classical music, can be beneficial for increasing focus and can improve learning outcomes (Angwin et al., 2017; Chou, 2010; Davies, 2000; Gao et al., 2020). Therefore, one interesting question emerges which is: what are the specific properties of sounds that affect human focus levels the most? Additionally, studies have shown that the effect of sounds is often subjective, where whether one likes a given sound or not is a key factor in its effect on their experience (Cassidy & Macdonald, 2009; Huang & Shih, 2011; Mori et al., 2014). Although this finding about the subjectivity of sound reappears across many studies, psychophysical thresholds are known to exist and there are clearly natural laws governing much of the way humans hear and experience sound (Levitin et al., 2012; Nia et al., 2015; Washburne, 2020). The potential of sounds to increase focus and demand for non-pharmaceutical tools that enable individuals to enhance their ability to focus has recently led several companies (including Endel, Brain.fm, Mubert, Enophone, Focus@Will, Melodia, AIVA, and others) to develop soundscapes that are dedicated to increasing focus on-demand. These soundscapes include elements of white noise, music, and other sonic properties that are functionally combined to increase a listener's focus and maintain high levels of focus over long durations of time. One of the challenges in this field is to figure out the physical properties of sound that contribute to human experience the most so that design principles can be defined correctly to create soundscapes that achieve the goal of increasing focus, opposed to the inverse of causing distractions and hurting an individual's ability to focus. Insights about sound properties therefore have been sought by commercial groups alongside academic groups in order to learn how to optimize experiences through sound. Many scientific studies have explored this question and looked for the relationship between sound, music and human experience using objective measures that empirically assess properties of audio and their emotional correlates. For example, Cheung et al (Cheung et al., 2019) found that pleasure from music depends on states of expectation, such as a skipped rhythmic beat, which can either be pleasurable or discomforting depending on the listener's circumstance. Sweet Anticipation (Huron, 2006) similarly maps how music evokes emotions within a theory of expectation and describes psychological mechanisms that are responsible for many people's mixed responses to sounds. Other studies used machine learning methods to map from features of audio signals to emotions (Brotzer et al., 2019; Cunningham et al., 2020; Hizlisoy et al., 2021; Vempala & Russo, 2012; Yang et al., 2008). These machine learning studies to date have, however, only aimed to predict emotions based on the limited valence-arousal circumplex model, and as far as we know, no attempts have been made to predict human focus levels exclusively based on audio signal analysis. One persistent obstacle to the field's understanding has been studies that rely on data with a low temporal resolution. Since sounds and emotions evolve fast, on the order of tens of milliseconds, the current lack of modeling tools capable of capturing the fast changes in human experience that accompany changes in sound is a major hindrance to progress (Cowen & Keltner, 2017; Larsen & Diener, 1992). Commonly, for example, reports are based on data where there is a single emotional label per song, while the song lasts ~2-3 minutes and throughout it there are emotional dynamics that change dramatically. This mismatch of data can lead to conclusions being drawn from inadequately small amounts of samples, and worse than that, inaccurate emotional labels.

Attention and emotion decoding from brain signal

Brain decoding technology offers an exceptional opportunity to tackle this issue, since it enables us to get an estimation for the experience dynamics at the same time resolution as focus phenomena occur. Using electroencephalogram (EEG) sensor data, which contains electrical brain activity measured from the scalp (non-invasive) on the order of hundreds of measurements per second, many studies have established that it is possible to capture fast changes in human emotions and experience, such as stress (Perez-Valero et al., 2021), arousal (Faller et al., 2019), fatigue (Hu, 2017), and happiness (Lin et al., 2017). Several studies have similarly shown the ability to capture focus and attentional state changes, affirming that this information as well is captured in EEG sensor data (Hamadicharef et al., 2009; Jung et al., 1997; Micoulaud-Franchi et al., 2014; Tuckute et al., 2021). While brain decoding technology has been applied widely to study the effects of different types of stimuli (e.g visual, tactile, auditory) on human experience (Asif et al., 2019; Bhatti et al., 2016; Shahabi & Moghimi, 2016), as far as we know, it has not been applied to study the joint effects of sound and focus at the high temporal resolution needed to explain both phenomena. In recent years, progress in the development of non-clinical, wearable EEG sensors (such as Muse, Neurosky, Emotiv, Bitbrain, etc.), which are intended for consumer uses, has led to new research paradigms where comfortable, affordable, wireless, and easy-to-use at-home measurement devices collect neuroscientific data "in-the-wild" at a large scale and make it possible for the first time to measure brain responses from diverse audiences within their natural, real-world environment. Many of the wearable devices offer decoding outputs beyond the raw sensor data, and these "off-the-shelf" decoding outputs include attention, relaxation, and other states (Abiri et al., 2019; Bird et al., 2019; González et al., 2015; Liu et al., 2013; Rebolledo-Mendez et al., 2009). It is important to note, however, that although decoder algorithms exist in the market for consumer uses, verifying their reliability to accurately capture attention, valence, arousal, stress and other attributes of human experience at a high temporal resolution and research quality has remained a challenge.

Combining brain decoding with sound tests to increase focus

In the current study, we used Arctop's brain decoding technology (neuOSTM) on data from portable EEG (Muse-S) headbands to measure human focus levels in individuals performing tasks at home while listening to different types of sounds. Since this is a relatively new decoding technology, we first evaluate the validity of the focus outputs within the experimental conditions. Then, once convinced of the output's veracity and reliability, we use the focus data to compare effects of different sound stimuli on individuals while performing different tasks. Next, we exploit the decoded data's high time resolution to map between raw audio signals and the focus dynamics. Based on this mapping, we build a model that takes sound properties and predicts human focus levels, enabling us to compare between new songs, sounds, and between genres to gain additional insights about the nature of sounds which drives human focus the most. These insights can help in the future to generate optimal playlists to increase focus, engineer better soundscapes, and even adapt sounds in real-time based on an individual's focus levels to enable them to precisely influence their own mental state.

Materials and Methods

Participants

Sixty-two (62) participants (40 males, 22 females, 18-65 years), completed four (4) sessions over a single (1) week at their own home. All participants were recruited from an opt-in screening panel and were distributed across the five (5) major regions of the continental United States (Northeast, Southwest, West, Southeast, and Midwest). Only participants who reported normal hearing, normal vision, or vision that was corrected to normal with contact lenses, were included. We excluded volunteers who reported using medication that might influence the experiment and who reported neurological or psychiatric conditions that could influence the results. Participants were native English speakers and a written informed consent was obtained from each participant prior to their participation. Participants received compensation for their time.

Paradigm

Tasks

Participants performed various tasks within a mobile app (neuOSTM by Arctop Inc.) while listening to one of three types of sound and wearing a brain signal measuring headband (4-channel EEG Muse-S device by Interaxon Inc.). Each participant received a kit at their home that included all the equipment needed to participate, including over-ear (Sony) headphones, headband and tablet computer with the mobile app installed. Participants recorded four one hour long sessions, while listening to different types of sounds. Sessions included 30 minutes of a "Preferred Task" — a task chosen by the participant — followed by short tasks used to validate the brain decoding outputs for each session. These validation tasks included video games (Tetris), math problems (Arithmetics), and word problems (Creativity). Participants were assigned to groups according to a pseudorandom schedule that controlled for potential sequence effects of the tasks and different sound stimulus types (Fig. 1). The short tasks were used to calibrate the Arctop decoding algorithms to a validated performance level, and afterwards the validated model was used to measure each participant's focus level across the Preferred Task.
Figure 1. Schematic illustration of the paradigm in each recording session.
Each session started with 30 minutes of a task selected by the participant (\"Preferred Task\"), followed by 3 minutes of arithmetics exercises, 3 minutes of a creativity task, and two levels of Tetris the video game (each level lasted 1 minute regardless of performance). After each task, participants answered a survey where they reported on aspects of their experience (e.g. focus, enjoyment, stress) using linear scale sliders from \"Not at all\" (0) to \"Very\" (1)."} /> Participants were instructed to choose a Preferred Task they could perform in a seated position while listening to sounds through the headphones, and which they would be happy to repeat in all four sessions. For example, Preferred Tasks that were chosen included knitting, working, reading, and solving Sudoku puzzles. At the end of each task the participants self-reported their experience through a survey in the app which used linearly-scaled slider buttons to quantify experience along several dimensions (e.g. focus level, enjoyment, stress, motivation, etc.). For the Preferred Task, the survey included reporting on their focus level during the first and second half of the task separately, resulting in six (6) self-reported quantitative focus labels per session (Preferred Task: 2 labels, arithmetics: 1 label, creativity: 1 label, tetris: 2 labels).

Sounds

Each participant experienced four sound conditions over the four days of the study: two music playlists by leading digital service providers Spotify and Apple (downloaded September 2020), one personalized soundscape engineered by Endel, and silence (no audible sounds). We selected Spotify's 'Focus Flow' playlist and Apple Music's 'Pure Focus' playlist to represent the category of pre-recorded sounds designed to increase focus. For soundscapes we selected the mobile application Endel to represent the category of real-time, engineered sounds that contain a mixture of noise and musical properties. The Endel app 'Focus' soundscape was used by each participant on their own device. All sound conditions were instrumental (i.e. did not include singing or any audible lyrics). For the condition of silence, participants wore headphones exactly as they did in the sound conditions, but no music or audible sounds of any kind were played and no soundscape was generated - participants simply completed the session in a quiet environment.

Data processing

Data acquisition

While participants were listening to sounds and engaging in the experimental tasks, their electrical brain activity was recorded using a portable, noninvasive electroencephalograph (EEG) headband that weighed 41 grams (Muse-S device by Interaxon Inc). The headband included four dry fabric EEG sensors (sampling rate: 256 Hz), photoplethysmography (PPG) sensors (for heart rate) and motion sensors (gyroscope, accelerometer). The brain-measuring EEG sensors are located on the scalp at two frontal channels (AF7, AF8) and two temporal channels (TP9, TP10), with the reference channel at Fpz. The headbands were put on by participants themselves with the assistance of a quality control screen that started each session by giving participants real-time feedback on the signal quality and made it easy to adjust the headband appropriately to acquire an optimal signal quality (Fig. 2). No technicians or other support staff assisted in the placement of the headbands - the process was completely automated by the in-app prompts, freeing the participants to complete sessions at any time of their choosing.
Figure 2. Schematic illustration of the data processing pipeline.
Data acquisition included at-home recordings of four sessions, each with a different background sound type. Arctop's neuOS brain decoding technology was used to predict the focus dynamics at a rate of 5Hz. Obtaining the brain decoded focus dynamics synchronously with the sound content enables comparison of focus levels correlated with different physical properties of sound."} />

Brain data based models of focus

Arctop brain decoding technology (neuOS) was used to transform the sensor data into predicted focus dynamics with a time resolution of 5Hz (Fig 2). For each participant, short tasks (games, word and math problems) were used to calibrate and validate a model of their focus based on the brain data, and then once validated the model was applied to the Preferred Task data. Fig. 3 shows the resulting brain decoded focus levels of two representative participants across all four sessions during the Preferred Task. Model performance was evaluated using Pearson correlation coefficient between the self-reported focus and the brain decoded focus values, and after thresholding the values, with the area under the ROC curve for binary classification of low/high focus (Fig. 5). Eleven (11) participants were excluded from further analysis following model validation due to excessive noise in their recorded brain data and/or unreliable survey responses, leaving a total of 51 participants (mean age= 36, SD=8, 17 females and 34 males) in the experimental analysis.
Figure 3. Brain data based focus model dynamics of two representative participants during the Preferred Task performed at each of the four sessions. Each row represents a session with a different sound stream playing in the background as participants perform their chosen task. Each session included 30 minutes (X axis = time in minutes) of a \"Preferred Task\" over which their focus level (Y axis = decoded focus) was measured. Participant 29 (A) was reading while Participant 45 (B) was working."} />

Statistical methods

For comparisons between average focus levels during the different sound content presented, we calculated for each participant (N=51) the median focus level while performing the preferred task and conducted a one-way repeated measures ANOVA (Analysis of variance) test. Then, if p<0.05, paired t-tests were applied post hoc to compare between pairs of sound streams using Holm-Bonferroni correction. Time series statistical tests were applied to compare focus level dynamics and discover specific time periods of significant difference. A paired t-test was applied at each second between focus levels of two sound streams. The p-values were then corrected for multiple comparisons by setting a threshold for a minimum significant sequential time-samples. The threshold was determined by random permutations (1000 iterations) of participants' conditions and repeating the statistical test, resulting in a distribution of significant sequential time samples. The threshold was set as the 95% percentile of the resultant distribution (Broday-Dvir et al., 2018).

Sound signal decomposition and feature extraction

The pre-recorded playlists conditions (Apple and Spotify) provided raw sound data that we used to obtain sound property dynamics in the time and frequency domain that could be correlated with the obtained focus dynamics. Soundscapes were not used in this analysis because they were produced in real-time personally for each participant, which limited the ability to apply sound property analysis appropriately across the data set. The sound features were calculated using Python's library pyAudioAnalysis (Giannakopoulos, 2015), for example, the sound signal energy, spectral entropy, and chroma coefficients. The features were calculated in short-time windows of 50 ms with a sliding window of 25 ms. Then, basic statistics were calculated over the sound features in windows of 30 seconds (e.g. mean and std), resulting in 136 sound properties (link to full list). To enable mapping to the brain model, the brain decoded focus levels were also averaged in the corresponding 30 seconds windows (Fig. 4).
Figure 4. Diagram demonstrating the framework for correlation of time-series focus values with sound properties. (A) Example of a recorded brain data in microvolts (single channel of EEG) segment, which after applying the preprocessing and trained models on 30 minutes of recordings, transforms to the brain decoded focus dynamics (top (C)). (B) Examples of a sound segment in decibels taken from one of the songs. (C) The sound features (y-axis) dynamics during 30 minutes of recordings (x-axis)."} /> To obtain the threshold for significant correlations between sound features and focus levels (p<0.05), a shuffle analysis was performed. Random permutations (1000 iterations) of the brain decoded focus levels were applied across songs to preserve the time dependency of focus levels within a song and the focus levels distribution. The correlation of each sound feature was calculated with the permuted focus level. The threshold was set as the 95% percentile of the resulting correlation's distribution.

Obtaining the sound decoded focus model

To map the relationship between properties of the sounds heard and focus levels measured from the brain, we first applied principal component analysis (PCA) to reduce the dimensionality of the sound features. We then trained regression models between the transformed sound features and the brain decoded focus through a 5-fold cross validation procedure that used 80% of the songs in each iteration to train and 20% to test. The presented sound decoded focus model is a linear model based on the first PCA component of the features (shifted and rescaled).

Results

Brain-measured focus levels accurately reflect self-reported focus levels

Before comparing focus levels elicited by the different sound types, we validated the underlying brain decoding technology by comparing between the brain-based focus predictions and the self-reported focus values. Figure 5A shows a histogram of the model performance per participant. The model is evaluated based on the AUC score (of the ROC curve) for prediction of self-reported focus during the Preferred Task (low-high focus) where the chance guessing level is = 0.5 (black dashed line). The average result across participants obtained was <AUC>=0.83 (N=51, SD=0.19), a strong validation of the brain-measured focus model's accuracy.
Figure 5. Validation of focus measurements derived from brain data.
(A) Histogram of focus models performance per participant (N=51), evaluated using the area under the ROC curve (AUC-ROC). Black dashed line marks chance level (0.5). (B) Average focus levels per event vs. self-reported focus resulted in Pearson correlation of 0.6. Inset shows ROC curves for different values of self-report threshold. (C) Confusion matrix after thresholding the focus score predictions and self-report. Classification scores for 2-classes (low focus vs. high focus) are AUC=0.87 (area under ROC curve), Accuracy=0.8. (D) Average brain decoded focus levels vs. average self-reported focus across the four sound types."} /> When aggregating the tasks from all participants, the Pearson correlation between the brain decoded focus model and the self-reported focus was Corr(414)=0.6, p<10−4 (Fig. 5B). The inset in Figure 5B shows the ROC curves for different values of self-reported threshold and the confusion matrix for one of these thresholds (0.4) resulted in an accuracy score of 0.8 (Fig. 5C). Figure 5D shows the average brain decoded focus level per sound type vs. the average self-reported score.

Soundscapes induce a higher focus level compared to silence

Using the validated focus models which output five measurements per second (5Hz), we then compared between the average focus levels elicited by the sounds during the Preferred Task. The background sound was found to have a significant effect (top row in Table 1, F(3,150)=4.28, p=0.006, statistical methods for details) on the elicited focus level, and the post hoc tests (Holm-Bonferroni correction) revealed that streaming soundscapes (with Endel app) was significantly higher compared to silence (Fig. 6A1, supp. Table 1, M=0.090, SE=0.027, t(50)=-3.38, p=0.008), while streaming music using Apple or Spotify did not have an effect (Apple: t(50)=-2.37, p=0.11, Spotify: t(50)=-1.24, p=0.65). For 35.3% of the participants the Endel session produced their highest focus level, while for 27.5% of participants the Apple playlist produced the highest focus level. For 19.6% of participants Spotify was best for producing focus and for 17.6% silence was (Fig. 6A2, the details sorted focus levels per participant are shown in Supp. Fig. 1).
Table 1. Results of a one-way repeated measures ANOVA performed on each subgroup comparing the average brain decoded focus levels of each sound stream during the Preferred Task. Sound most significantly affected those below 36 years old."} />
Figure 6. Comparison of the brain decoded focus during the Preferred Task while listening to different sounds.
Top row - Average focus levels for each sound stream during the Preferred Task for each group of interest, including statistical results. Error bars are standard errors. Bottom row - Distribution of the best session (highest focus on average) for each participant per group. The groups of interest are: (A) All participants (51), (B) Participants who were working during the Preferred Task (26), (C) Participants who were not working (25 -reading, knitting, playing, etc). (D) Participants above 36 (26). (E) Participants below 36 (25)."} /> To gain a better understanding of the conditions where sound affected focus, we next split the participants into subgroups of interest and repeated the statistical analysis. We first asked whether the focus level difference is task dependent. During the Preferred Task, 51% of the participants (26) chose to work, while the rest (49%) read a book (29.4%), played games (9.8%) or did other various tasks (e.g. knitting, 9.8%). To assess the effect of sounds on focus levels during these different tasks, we split the participants to the ones who worked and those that did other tasks. We found that for the "working" group, the focus level elicited by Endel's soundscapes was higher compared to silence (M=0.12, SE=0.04, t(25)=3.26, p=0.017), while for the "not-working" group there was no difference (Fig. 6, supp. Table 1, M=0.06, SE=0.04, t(24)=1.552, p=0.447). These results suggest that the focus level differences between Endel and Silence are task-dependent, where the sound is beneficial for specific types of tasks, namely, "working." We next split the participants into two age groups according to the median age (36 years). We found that for the younger participants (age<36, N=25), all sound types were superior to silence for producing elevated focus levels (Fig. 6, supp. Table 1, M=0.14,0.13,0.12, SE=0.04,0.03,0.03, t(24)=3.79,4.49,3.67, p=0.004,0.001,0.005 for Endel, Apple and Spotify respectively) while for the older participants (age>36, N=26), there was no difference between sound and silence. The focus level differences were therefore found to also be age-dependent.

Time series analysis of the focus dynamics reveal differences between all sound types and silence

Exploiting the high temporal resolution of the focus measurements, we compared the focus dynamics to each sound stream that played during the 30 minutes of the Preferred Task (Fig. 7, table 2). When comparing Endel's soundscapes vs. Silence (Fig. 7A), we found that the focus level elicited by Endel's soundscape was higher 87% of the time, a separation whose significance started after 2.5 minutes of listening. In addition, although on average there wasn't a significant difference, the focus level elicited by Apple's playlist was higher than Silence 60% of the time, starting at 12.5 minutes (Fig. 7C), and the focus level elicited by Spotify's playlist was higher than Silence 27% of the time, starting at 17 minutes (Fig. 7B). Focus elicited by Endel's soundscape was higher than Spotify's playlist in 37% of the time, starting at 6 minutes (Fig. 7D).
Figure 7. Comparing brain decoded focus dynamics during the 30 minutes of the Preferred Task.
Each subfigure shows a comparison between two sound streams, while the gray areas are the timings with a significant difference (p<0.05 corrected, see statistical methods for details)."} />
Table 2. Summary of focus time dynamics comparison, showing for each pair the percentage of time and time segments with significant difference (where 100% = 30 minutes)."} />

Focus levels in response to sounds can be predicted by the sound's properties

Seeing that background sound had an effect on focus levels, we go further and ask whether music and soundscapes can be composed according to a formula to increase focus levels. Meaning, can we understand which sound properties drive focus well enough to predict focus levels from only an analysis of the sound properties themselves? Leveraging the high temporal resolution of the noninvasive brain measurements, we generated a prediction model which predicts the brain-based focus level from sound features extracted from the audio signal. Raw audio files containing the Apple and Spotify sessions were used to extract different sound properties with a running sliding window of 30 seconds. The personalized soundscape session (Endel) was not used in this analysis since the real time streaming did not allow saving the raw audio files that were consistent across participants. Each sound property was treated as a unique feature and checked for its contributory power to the measured average focus level. Supp. Fig. 2 shows the resulting correlations between each sound feature and the brain based focus level. In total, only 20 features of the 136 features evaluated were found to have significant correlations (p<0.05, see Statistical methods). We next combined multiple sound features to generate a sound data based model that predicts focus levels (see Methods). Figure 8 shows the dynamics of the focus predicted from the audio signal alone which included only properties of the sounds, together with the brain decoded focus that was derived from high resolution electrical brain measurements (Corr=0.7, p<1e-4). Figure 8D shows that if we threshold our dynamics to output a binary prediction (low/high focus), the audio model reaches 88% accuracy in predicting the brain based focus (AUC=0.93).
Figure 8. Results of predicting brain decoded focus from audio features.
(A+B) Dynamics of brain decoded focus (dark blue) and audio decoded focus (light blue), during 30 minutes of the Preferred Task for Apple (A) and Spotify (B). (C) Brain decoded focus (y-axis) vs. Audio decoded focus (x-axis) for both playlists (Apple + Spotify). (D) Confusion matrix after thresholding the focus predictions to classify between low and high focus. Classification accuracy obtained: 88% (Area under ROC curve: 0.93)."} /> Beyond composing soundscapes for focus, we can also use these prediction models to rate the focus level of a song and assemble successful playlists based on existing songs. To demonstrate this, we compared the song average of the audio decoded output to the brain decoded output. As shown in Figure 9B, there is a correlation of 0.74 between the focus models at the song level (df=16, p=0.0004). Figure 9A shows these averages sorted by the brain-based model.
Figure 9. Averaging focus scores for each song.
(A) Sorted focus scores per song obtained by the brain model (brain decoded - blue), next to the focus obtained by the audio model (audio decoded- light blue).
‍(B) Focus scores per song - brain decoded (y axis) vs. audio decoded (x axis). Pearson correlation between them: Corr(16)=0.74, p=0.0004."} />

Analysis of a sound's properties can be used to predict its effect on focus

To gain additional insights about the effects of different sound types on human focus, we used the trained audio model, and projected songs and sounds which were not played during the brain recording experiment. Meaning, we obtained their focus score and dynamics based solely on the properties of the sounds they contained. We selected sounds that challenged the validity of the audio model based on their categorical exclusion from the brain recording experiment. A future approach can include these different genres as controls for further brain measurement validation studies. For example, Endel's soundscapes which are not personalized (taken from the playlist: "Focus: Calm Clear Morning"), natural sounds which are commonly used for increasing focus (such as white noise, waves, rain, taken from: https://mc2method.org/white-noise/), and popular songs from other music genres (classical music, electronic, pop, rock, jazz and hip hop) were used. Figure 10A shows the predicted focus score based on the audio model which took into account only the properties of the sounds themselves. Songs are sorted from the highest focus evoking song (Endel - Three No Paradoxes) to the lowest (Dr. Dre - What's The Difference). The top two songs are Endel soundscapes which are not personalized, a finding which strengthens our main result since it implies that the high focus scores elicited by Endel's soundscape was not a byproduct of personalization. Figure 10B shows the sorted focus scores averaged across genres, where notably sounds from classical music and natural sounds contained properties that predicted high focus levels. In contrast, pop and hip-hop songs predicted relatively low focus scores. Although we do not have ground truth focus labels for these songs based on real human brain data, given the relatively high scores of the sounds which are known to have generated increased focus in the experimental data, we can conclude that there is a consistent validity to the model. Future research will gather ground truth labels for these songs and evaluate the model mathematically in this context.
Figure 10. Projecting new songs into the trained audio model.
(A) Sorted focus scores per song obtained by the audio model, colored by genre. (B) Average focus score per genre, sorted from the genre with the lowest score (Hip-hop) to the highest (Classical). (C) Average focus variance per genre, sorted from the genre with the lowest variance (Natural) to highest (Electronic). (D-E) Focus dynamics for the song with the lowest focus score (D) and the highest (E)."} /> Analyzing the average within-song variance across different genres reveals that the model predicts the largest variance on average for electronic sounds (Fig. 10C), while the lowest variance was found for natural sounds. The variance can be interpreted as a range of focus dynamics, and indeed the focus dynamics of the electronic sounds might change dramatically during a song (supp. Fig3), confirming the need for a tool which outputs dynamics with a high temporal resolution when studying such sound content. Figure 10D-E shows the focus dynamics for the song with the lowest focus evoking score and the highest. The dynamics for all songs can be seen in supp. Fig.3.

Discussion

"The soundscape of the world is changing. Modern man is beginning to inhabit a world with an acoustical environment radically different from any he has hitherto known" said the composer R. Murray Schafer, presaging the time we live in now when the sounds available to us continue to multiply by the day. As we have an increasing number of options to modulate our auditory lives by, a handful of take-aways from this study standout:

Brain-based measurement of focus is possible "in-the-wild"

Although the effects of sound and music on the human brain can be subtle in measured brain signals when judging by the changes produced in raw electromagnetic currents, they are robust and highly quantifiable with effectively-trained algorithms, as shown here. Classifying emotional and attentional responses is particularly useful when done at the sub-second temporal resolution which allows one to track dynamics continuously over time at the same timescale as the brain functions that impact perception and behavior. In this study we demonstrated that noninvasive brain decoding technology is able to deliver this needed resolution with a high degree of accuracy (approximately 80% match to self-report, Fig 5C.). Since there are inherent biases in the subjective self-report for experience (Kahneman et al., 1999; Mauss & Robinson, 2009), when mapping physiological signals to self-reported experiences, as done here, there is an upper boundary for accuracy beyond which the model will over fit to the self-reported values and incorrectly represent the information observed in physiological signals. According to a recent review (Larradet et al., 2020) which summarizes multiple peer-reviewed studies that predict self-reported emotions from physiological signals, the average accuracy reported was ~82%. Given this average and the experimental conditions here (small amount of sensors, at home recordings, simple self-report scales), the achieved accuracy was satisfactory for drawing deeper conclusions on sound properties, and aligned with state-of-the-art emotion recognition accuracies in the context of sound as a stimulus (Tripathi et al., 2017, 81.41% and 73.35% for 2 classes of Valence and Arousal respectively). A key benefit of the current approach is that this method of high temporal resolution brain measurement can be performed reliably outside of traditional laboratories. In this current study not a single laboratory or facility was used for data acquisition. Instead, 18-65 year olds across the U.S. received a technology kit in the mail and experienced music playlists and personalized soundscapes while they recorded their own brain signals from the comfort of their own homes at times of day of their choosing. In other words, in their natural habitat, at their own pace, which lends the research a rare ecological validity.

Focus is increased most by personalized, engineered soundscapes

Within the at-home environment of this study, personalized, engineered soundscapes were found to be the best at increasing participant's focus levels (Fig. 6A). After 2.5 minutes, on average, listeners of the personalized soundscapes evaluated experienced a meaningful increase in their focus level, while for music playlists it took approximately 15 minutes to gain a similarly appreciable increase (Fig. 7). The audio effect on focus levels was found to be task dependent, where soundscapes increased focus levels in participants who were working (Fig 6B). For participants who were not working, no significant difference was found. This result suggests that willful orientation of attention towards work tasks may have created a brain context especially suited to modification by sound. While engaged in work, participants may also have been more prone to distraction and thus more impacted by the positive uplift of sound compared to reading or playing a game which may have contained more intrinsic motivation to stay focused on. One limitation of this current study is that it did not allow us to disentangle the effects of personalization of sounds on the listener, since pre-recorded soundscapes were not tested. Equivalently, a comparison of personalized soundscapes to personalized music playlists, where audiences either made their own playlist for focus or were allowed to skip songs whenever they wanted, will likely contribute to a fuller understanding of how sound properties correlate with emotion and attention changes. Follow-up research will incorporate these variables. An additional limitation was the inability to reach conclusions regarding gender-dependent effects which was at least partially due to this study's imbalanced data set. Despite efforts to recruit a balanced group of participants, enrollment was done on a rolling basis and in the end the female subgroup was statistically underpowered. In future research, especially for closed loop, real-time testing, balanced participant sets will be important for reaching more detailed conclusions.

Sound preferences and focus effects vary between people

It is important to emphasize that the results reported here are sound effects on the average focus levels across a population, and that there was a large variance in this effect between participants. Evidence for this large variance can be seen in Supp Fig. 3 and in the age dependency effect (Fig 6D-E), where for the younger audience, all sounds increased focus while for the older audience, the sounds did not have any effect. These results are consistent with other studies showing personal preferences are critical for the improvements possible by sounds (Cassidy & Macdonald, 2009; Huang & Shih, 2011; Mori et al., 2014). Due to this variety observed together with the highest focus being elicited by the personalized soundscapes, a next step will include closed-loop selections of sounds, where iterative sound testing is used per person to identify the significant parameters for maximizing focus for that person. Personalized soundscapes specifically, and personalized audio in general, should be investigated further for their capacity to increase productivity, creativity and well-being as these attributes of human experience are associated with one's ability to focus. For clinical populations as well, for example children with ADHD, the tailoring of sounds for this purpose of increased focus can be particularly impactful. It is possible that the seamlessness of the personalized soundscapes tested here, which played continuously without gaps in the sound like the music playlists had between songs, was a critical part of the observed effect on focus. At every juncture of the experience there is more to be learned, but at a high level, a main lesson of this study is that there is a strong need for personalization of sound in order to most effectively achieve functional goals like increasing focus.

Brain decoded focus data enabled a new predictive model based on sound data alone

Leveraging the high temporal resolution of the brain decoded dynamics, a focus prediction model based on the physical properties of sounds was successfully trained, resulting in an accuracy score of 88% in predicting the brain decoded focus score from an audio decomposition that assessed 136 different properties of sounds as unique features (Fig. 8). This model enabled a further examination of how sounds and different genres effects focus and allowed testing additional conditions, such as pre-recorded soundscapes and commonly used background sounds (e.g. white noise), as well as other genres (pop, rock, jazz, etc). We found that the model predicted the highest focus score to classical music, followed by engineered soundscapes and natural sounds. These results complement previous studies which showed natural sounds and classical music are beneficial for learning and concentration (Angwin et al., 2017; Davies, 2000; DeLoach et al., 2015; H. Liu et al., 2021). In contrast, the models predicted that genres such as pop and hip-hop produce lower focus scores (Fig 10A-B). It is possible that these sounds contain more distractors that attract attention away from other objects of attention, or that they contain types of sounds that the brain requires more resources to process (depending on familiar patterns, surprises and more), leading to less resources available to perform other tasks. Sounds in these genres may also activate the reward system more (Gold et al., 2019; Salimpoor et al., 2015), which can increase motivation and improve learning of the songs themselves rather than orientation towards other tasks. These sounds may be optimal for driving focus on the songs in other words, rather than focus on other things. Understanding the brain mechanisms that underlie the modified focus is beyond the scope of this work, but the mapping found here can potentially provide fruitful directions for future brain imaging experiments that are equipped to answer these questions. The analysis here demonstrates a process in which we utilize the temporal resolution of the brain sensing technology to generate a product where the neurotechnology is eventually out of the loop, resulting in a stand alone sound model which gets as an input a raw audio file and outputs a predicted focus score. This model can be used independently to generate focus playlists or to compose optimal soundscapes, and can further be improved by expanding to populations outside the U.S. and different age groups. Pythagoras, who first identified the mathematical connection between a string's length and it's pitch, believed that the whole cosmos was a form of musical composition (James, 1995). We too see the rich mathematical models obtained in this study, by mapping sound properties to human experience, as a glimpse into the natural laws governing how we feel and think. The better these laws can be understood, the more empowered individuals will be to modulate their sound environments to suit their goals and states of mind. There remains much to figure out. While we as a species continue to cause a "shift in the sensorium," we simultaneously experience that shift all over daily life and it is not clear where we as a species are headed. This study showed that sounds have a distinct effect on our focus, and paves the way for designing sounds to help us focus better in the future.

Conclusions

We studied the effects of sound on human focus levels using noninvasive brain decoding technology and to gain a better understanding of the optimal sound properties for increasing focus levels in listeners. We combined a custom app, portable EEG-measuring headbands, and brain decoding technology that enabled us to obtain high temporal resolution focus dynamics from participants at home. Using the brain decoded focus dynamics, we then analyzed how various properties of sound affected focus levels in different tasks. We found that while performing a self-paced task for a long period of time (such as working), personalized soundscapes increased focus the most relative to silence. Curated playlists of pre-recorded songs by Apple and Spotify also increased focus during specific time intervals, especially for the youngest audience demographic. Large variance in response profiles across participants, together with task and age dependent effects, suggest that personalizing sounds in real-time may be the best strategy overall for producing focus in a given listener. Finally, we generated a sound property based focus model which successfully predicts the brain decoded focus scores using only an audio file as input. Using this model, we extracted predicted focus scores from new songs based on audio decomposition and performed a genre analysis to develop new intuitions about the findings and the source of focus-producing sound content. We found that based on our model, engineered soundscapes and classical music are the best for increasing focus, while pop and hip-hop music are the worst. The approach taken here can be adapted to include other emotions (e.g. enjoyment, stress, happiness, etc.), attentional parameters ('Flow state,' memory formation, etc.) and can be used to assess additional content as well (e.g. visual, ambient, olfactory, etc.), including interactive gaming and e-learning where personalization and high temporal resolution experience measures may be especially beneficial.

Data Availability Statement

The dataset for this study is available through an open Git repository (link). Data includes the brain decoded focus dynamics for each participant together with scripts that run the statistical tests.

Author Contributions

A.H., R.K, N.E, E.K, and D.F designed the experiment, A.H analyzed the data, R.K, N.E, and D.F. advised on data analysis and statistics. S.K and E.K developed the app and software platform for data collection. A.H and D.F wrote the paper, R.K and E.K revised the paper. All authors approved the work for publication.

Conflict of Interest

This study received funding from Arctop Inc. and Endel Sound GmbH. The funders had the following involvement with the study: Arctop Inc. was involved in study design, collection, analysis, interpretation of data, the writing of this article and the decision to submit it for publication. Endel Sound GmbH was involved in study design and provided audio stimuli used in the experiment. All authors declare no other competing interests.

Acknowledgments

We would like to thank Hillel Pratt, Kevin Liu, Alexander Kopanev, Warner Music, Sony Music, Endel, and Universal Music for providing audio content, data, and support in conducting this study and advancing theoretical and applied aspects of the research.

Funding

This study received funding from Arctop Inc. and Endel Sound GmbH. Endel Sound GmbH was not involved in data collection, analysis, interpretation of data, the writing of this article or the decision to submit it for publication. All authors declare no other competing interests. Complexity, 2019, e6862031. https://doi.org/10.1155/2019/6862031"},{"html":"Angwin, A. J., Wilson, W. J., Arnott, W. L., Signorini, A., Barry, R. J., & Copland, D. A. (2017). White noise enhances new-word learning in healthy adults. Scientific Reports, 7(1), 13045. https://doi.org/10.1038/s41598-017-13383-3"},{"html":"Asif, A., Majid, M., & Anwar, S. M. (2019). Human stress classification using EEG signals in response to music tracks. Computers in Biology and Medicine, 107, 182–196."},{"html":"Bhatti, A. M., Majid, M., Anwar, S. M., & Khan, B. (2016). Human emotion recognition and analysis in response to audio music using brain signals. Computers in Human Behavior, 65, 267–275. https://doi.org/10.1016/j.chb.2016.08.029"},{"html":"Bird, J. J., Ekart, A., Buckingham, C. D., & Faria, D. R. (2019). Mental emotional sentiment classification with an eeg-based brain-machine interface. Proceedings of TheInternational Conference on Digital Image and Signal Processing (DISP'19)."},{"html":"Broday-Dvir, R., Grossman, S., Furman-Haran, E., & Malach, R. (2018). Quenching of spontaneous fluctuations by attention in human visual cortex. NeuroImage, 171, 84–98. https://doi.org/10.1016/j.neuroimage.2017.12.089"},{"html":"Brotzer, J. M., Mosqueda, E. R., & Gorro, K. (2019). Predicting emotion in music through audio pattern analysis. IOP Conference Series: Materials Science and Engineering, 482, 012021. https://doi.org/10.1088/1757-899X/482/1/012021"},{"html":"Cassidy, G., & Macdonald, R. (2009). The effects of music choice on task performance: A study of the impact of self-selected and experimenter-selected music on driving game performance and experience. Musicae Scientiae, 13(2), 357–386. https://doi.org/10.1177/102986490901300207"},{"html":"Chanda, M. L., & Levitin, D. J. (2013). The neurochemistry of music. Trends in Cognitive Sciences, 17(4), 179–193. https://doi.org/10.1016/j.tics.2013.02.007"},{"html":"Cheung, V. K. M., Harrison, P. M. C., Meyer, L., Pearce, M. T., Haynes, J.-D., & Koelsch, S. (2019). Uncertainty and Surprise Jointly Predict Musical Pleasure and Amygdala, Hippocampus, and Auditory Cortex Activity. Current Biology, 29(23), 4084–4092.e4. https://doi.org/10.1016/j.cub.2019.09.067"},{"html":"Chou, P. T.-M. (2010). Attention Drainage Effect: How Background Music Effects Concentration in Taiwanese College Students. Journal of the Scholarship of Teaching and Learning, 10(1), 36–46."},{"html":"Cowen, A. S., & Keltner, D. (2017). Self-report captures 27 distinct categories of emotion bridged by continuous gradients. Proceedings of the National Academy of Sciences, 114(38), E7900–E7909."},{"html":"Cunningham, S., Ridley, H., Weinel, J., & Picking, R. (2020). Supervised machine learning for audio emotion recognition. Personal and Ubiquitous Computing, 1–14."},{"html":"Davies, M. A. (2000). Learning … the Beat Goes on. Childhood Education, 76(3), 148–153. https://doi.org/10.1080/00094056.2000.10522096"},{"html":"Davis, W. B., & Thaut, M. H. (1989). The Influence of Preferred Relaxing Music on Measures of State Anxiety, Relaxation, and Physiological Responses 1. Journal of Music Therapy, 26(4), 168–187. https://doi.org/10.1093/jmt/26.4.168"},{"html":"de la Mora Velasco, E., & Hirumi, A. (2020). The effects of background music on learning: A systematic review of literature to guide future research and practice. Educational Technology Research and Development, 68(6), 2817–2837. https://doi.org/10.1007/s11423-020-09783-4"},{"html":"DeLoach, A. G., Carter, J. P., & Braasch, J. (2015). Tuning the cognitive environment: Sound masking with “natural” sounds in open-plan offices. The Journal of the Acoustical Society of America, 137(4), 2291–2291. https://doi.org/10.1121/1.4920363"},{"html":"Faller, J., Cummings, J., Saproo, S., & Sajda, P. (2019). Regulation of arousal via online neurofeedback improves human performance in a demanding sensory-motor task. Proceedings of the National Academy of Sciences, 116(13), 6482–6490."},{"html":"Gao, C., Fillmore, P., & Scullin, M. K. (2020). Classical music, educational learning, and slow wave sleep: A targeted memory reactivation experiment. Neurobiology of Learning and Memory, 171, 107206. https://doi.org/10.1016/j.nlm.2020.107206"},{"html":"Giannakopoulos, T. (2015). pyAudioAnalysis: An Open-Source Python Library for Audio Signal Analysis. PLOS ONE, 10(12), e0144610. https://doi.org/10.1371/journal.pone.0144610"},{"html":"Gold, B. P., Mas-Herrero, E., Zeighami, Y., Benovoy, M., Dagher, A., & Zatorre, R. J. (2019). Musical reward prediction errors engage the nucleus accumbens and motivate learning. Proceedings of the National Academy of Sciences, 116(8), 3310–3315. https://doi.org/10.1073/pnas.1809855116"},{"html":"D. D. Schmorrow & C. M. Fidopiastis González, V. M., Robbes, R., Góngora, G., & Medina, S. (2015). Measuring Concentration While Programming with Low-Cost BCI Devices: Differences Between Debugging and Creativity Tasks. In D. D. Schmorrow & C. M. Fidopiastis (Eds.), Foundations of Augmented Cognition (pp. 605–615). Springer International Publishing. https://doi.org/10.1007/978-3-319-20816-9_58‍"},{"html":"Hallam, S., Price, J., & Katsarou, G. (2002). The Effects of Background Music on Primary School Pupils’ Task Performance. Educational Studies, 28(2), 111–122. https://doi.org/10.1080/03055690220124551‍"},{"html":"Hamadicharef, B., Zhang, H., Guan, C., Wang, C., Phua, K. S., Tee, K. P., & Ang, K. K. (2009). Learning EEG-based spectral-spatial patterns for attention level measurement. 2009 IEEE International Symposium on Circuits and Systems, 1465–1468."},{"html":"Hizlisoy, S., Yildirim, S., & Tufekci, Z. (2021). Music emotion recognition using convolutional long short term memory deep neural networks. Engineering Science and Technology, an International Journal, 24(3), 760–767. https://doi.org/10.1016/j.jestch.2020.10.009‍"},{"html":"Hu, J. (2017). Automated Detection of Driver Fatigue Based on AdaBoost Classifier with EEG Signals. Frontiers in Computational Neuroscience, 0. https://doi.org/10.3389/fncom.2017.00072‍"},{"html":"Huang, R.-H., & Shih, Y.-N. (2011). Effects of background music on concentration of workers. Work, 38(4), 383–387. https://doi.org/10.3233/WOR-2011-1141‍"},{"html":"Huron, D. B. (2006). Sweet Anticipation: Music and the Psychology of Expectation. MIT Press."},{"html":"James, J. (1995). The Music of the Spheres: Music, Science, and the Natural Order of the Universe. Copernicus. https://www.springer.com/gp/book/9780387944746‍"},{"html":"Jung, T.-P., Makeig, S., Stensmo, M., & Sejnowski, T. J. (1997). Estimating alertness from the EEG power spectrum. IEEE Transactions on Biomedical Engineering, 44(1), 60–69."},{"html":"Kahneman, D., Diener, E., & Schwarz, N. (1999). Well-being: Foundations of hedonic psychology. Russell Sage Foundation."},{"html":"Kumar, S., von Kriegstein, K., Friston, K., & Griffiths, T. D. (2012). Features versus feelings: Dissociable representations of the acoustic features and valence of aversive sounds. Journal of Neuroscience, 32(41), 14184–14192."},{"html":"Larradet, F., Niewiadomski, R., Barresi, G., Caldwell, D. G., & Mattos, L. S. (2020). Toward Emotion Recognition From Physiological Signals in the Wild: Approaching the Methodological Issues in Real-Life Data Collection. Frontiers in Psychology, 11, 1111. https://doi.org/10.3389/fpsyg.2020.01111‍"},{"html":"Larsen, R. J., & Diener, E. (1992). Promises and problems with the circumplex model of emotion.‍"},{"html":"Levitin, D. J. (2006). This is your brain on music: The science of a human obsession. Penguin."},{"html":"Levitin, D. J., Chordia, P., & Menon, V. (2012). Musical rhythm spectra from Bach to Joplin obey a 1/f power law. Proceedings of the National Academy of Sciences, 109(10), 3716–3720. https://doi.org/10.1073/pnas.1113828109‍"},{"html":"Lin, Y.-P., Jao, P.-K., & Yang, Y.-H. (2017). Improving Cross-Day EEG-Based Emotion Classification Using Robust Principal Component Analysis. Frontiers in Computational Neuroscience, 0. https://doi.org/10.3389/fncom.2017.00064‍"},{"html":"Liu, H., He, H., & Qin, J. (2021). Does background sounds distort concentration and verbal reasoning performance in open-plan office? Applied Acoustics, 172, 107577. https://doi.org/10.1016/j.apacoust.2020.107577‍"},{"html":"Liu, N.-H., Chiang, C.-Y., & Chu, H.-C. (2013). Recognizing the Degree of Human Attention Using EEG Signals from Mobile Sensors. Sensors (Basel, Switzerland), 13(8), 10273–10286. https://doi.org/10.3390/s130810273‍"},{"html":"Mauss, I. B., & Robinson, M. D. (2009). Measures of emotion: A review. Cognition and Emotion, 23(2), 209–237. https://doi.org/10.1080/02699930802204677‍"},{"html":"Micoulaud-Franchi, J.-A., Geoffroy, P. A., Fond, G., Lopez, R., Bioulac, S., & Philip, P. (2014). EEG neurofeedback treatments in children with ADHD: An updated meta-analysis of randomized controlled trials. Frontiers in Human Neuroscience, 0. https://doi.org/10.3389/fnhum.2014.00906‍"},{"html":"Mori, F., Naghsh, F. A., & Tezuka, T. (2014). The Effect of Music on the Level of Mental Concentration and its Temporal Change. Proceedings of the 6th International Conference on Computer Supported Education - Volume 1, 34–42. https://doi.org/10.5220/0004791100340042‍"},{"html":"Nia, H. T., Jain, A. D., Liu, Y., Alam, M.-R., Barnas, R., & Makris, N. C. (2015). The evolution of air resonance power efficiency in the violin and its ancestors. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, 471(2175), 20140905. https://doi.org/10.1098/rspa.2014.0905‍"},{"html":"Perez-Valero, E., Vaquero-Blasco, M. A., Lopez-Gordo, M. A., & Morillas, C. (2021). Quantitative Assessment of Stress Through EEG During a Virtual Reality Stress-Relax Session. Frontiers in Computational Neuroscience, 0. https://doi.org/10.3389/fncom.2021.684423‍"},{"html":"J. A. Jacko Rebolledo-Mendez, G., Dunwell, I., Martínez-Mirón, E. A., Vargas-Cerdán, M. D., de Freitas, S., Liarokapis, F., & García-Gaona, A. R. (2009). Assessing NeuroSky’s Usability to Detect Attention Levels in an Assessment Exercise. In J. A. Jacko (Ed.), Human-Computer Interaction. New Trends (pp. 149–158). Springer. https://doi.org/10.1007/978-3-642-02574-7_17‍"},{"html":"Sacks, O. (2010). Musicophilia: Tales of music and the brain. Vintage Canada."},{"html":"Salimpoor, V. N., Zald, D. H., Zatorre, R. J., Dagher, A., & McIntosh, A. R. (2015). Predictions and the brain: How musical sounds become rewarding. Trends in Cognitive Sciences, 19(2), 86–91. https://doi.org/10.1016/j.tics.2014.12.001‍"},{"html":"Schreiber, C. A., & Kahneman, D. (2000). Determinants of the remembered utility of aversive sounds. Journal of Experimental Psychology: General, 129(1), 27."},{"html":"Shahabi, H., & Moghimi, S. (2016). Toward automatic detection of brain responses to emotional music through analysis of EEG effective connectivity. Computers in Human Behavior, 58, 231–239. https://doi.org/10.1016/j.chb.2016.01.005‍"},{"html":"Shih, Y.-N., Huang, R.-H., & Chiang, H.-Y. (2012). Background music: Effects on attention performance. Work, 42(4), 573–578. https://doi.org/10.3233/WOR-2012-1410‍"},{"html":"Tripathi, S., Acharya, S., Sharma, R. D., Mittal, S., & Bhattacharya, S. (2017). Using Deep and Convolutional Neural Networks for Accurate Emotion Classification on DEAP Dataset. Twenty-Ninth IAAI Conference."},{"html":"Tuckute, G., Hansen, S. T., Kjaer, T. W., & Hansen, L. K. (2021). Real-Time Decoding of Attentional States Using Closed-Loop EEG Neurofeedback. Neural Computation, 33(4), 967–1004. https://doi.org/10.1162/neco_a_01363‍"},{"html":"Vempala, N. N., & Russo, F. A. (2012). Predicting emotion from music audio features using neural networks. Proceedings of the 9th International Symposium on Computer Music Modeling and Retrieval (CMMR), 336–343."},{"html":"Washburne, C. (2020). “More Cowbell”: Latin Jazz in the Twenty-First Century. In Latin Jazz. Oxford University Press. https://doi.org/10.1093/oso/9780195371628.003.0007‍"},{"html":"Yang, Y.-H., Lin, Y.-C., Su, Y.-F., & Chen, H. H. (2008). A Regression Approach to Music Emotion Recognition. IEEE Transactions on Audio, Speech, and Language Processing, 16(2), 448–457. https://doi.org/10.1109/TASL.2007.911513‍"},{"html":"Zald, D. H., & Pardo, J. V. (2002). The neural correlates of aversive auditory stimulation. Neuroimage, 16(3), 746–753."}]} />