This post is meant as an informal, quick transmission of a technical discovery to introduce our novel algorithm (FCCA) for mapping inter-subject neural synchrony to memory function. Fuller description of methods and discussion to follow in an upcoming version. The authors would like to thank collaborators Erez Simony, Aia Haruvi, Shai Kalev, Ronen Kopito for their contributions.
Abstract
Neural synchrony, similarities in brain activity across individuals, offers valuable insights into emotional and cognitive processing. It can also be used, under certain conditions, to infer subjective states of consciousness1. Objective metrics of similarities between people's brain data, or inter-subject correlations (ISC), we map here to dynamics of memory performance using a novel algorithm that builds upon the framework developed in correlated components analysis (CorrCA), generating a new higher-order analysis layer that obtains superior performance in memory decoding from EEG data compared to all other known methods. We call the method feature-based CorrCA (FCCA), and find it enables a robust, accurate memory analysis from people at home, in non-laboratory environments, involved in everyday tasks. By selectively deriving and using particular EEG activity features relevant to cognitive and emotional processes reflected in neural synchrony, FCCA serves as a powerful tool for investigation of the neural basis of shared experiences and highlights the nuance needed for accurate memory decoding from EEG. Validated on two models of consumer-grade EEG recording devices, our generalizable approach offers new possibilities for studying collective experiences in various contexts, from education to entertainment, and contributes to the study and understanding of social cognition and memory.
Episodic memory in particular has been an active area of research for modern neuroscience and there is a wealth of published findings in the literature relating to how physiological measures of brain activity during information encoding map to behaviorally demonstrable memory performance in the future. (Lee 2022)(Rugg, 2007, 2013), (Simony 2016), (Summerfield 2006). Studying hippocampus-cortex interaction during movie watching has been one fruitful experimental direction for the field. Along this line of inquiry we report on a naturalistic neuroscience study we conducted involving 60 adult participants distributed across the United States. Participants measured their EEG brain signal using a consumer wearable headband with sensors embedded while watching 38 short video clips and a 27-minute episode of the TV show 'Curb Your Enthusiasm' and days later were quizzed through multiple choice tests on aspects of the content to investigate whether FCCA can help elucidate relationships between memory formation and neural synchrony. We found that ISC dynamics derived by our FCCA algorithm can effectively capture memorability level dynamics while individuals experience the same stimulus.
Introduction
Earlier research has established a significant relationship between inter-subject neural synchrony and memory2,3. Both inter-brain synchrony and memory were separately found to be related to the emotional valence of events4–7. However, studies connecting the dynamics of these three processes to commonly experienced content are lacking. Traditional approaches have primarily relied on research-grade equipment and restrictive experimental paradigms, with limited research into the capabilities of consumer-grade EEG that are accessible for use at home nor the element of privacy and personal freedom that influences how participants experience TV shows at home versus in a social, focus group setting. This gap suggested to us the opportunity to develop an algorithm for populations in the real world experiencing the same content.
Recent advancements in neuroimaging techniques provide for a detailed exploration of neural synchrony, and the emergence of hyperscanning, the neuroimaging of multiple individuals simultaneously, has proven the utility of such algorithms for research and medical applications 8,9. This progress has highlighted the usefulness of robust brain synchrony algorithms and especially those that are suitable for real-time analysis with consumer-grade neural activity measurement devices that not only function effectively with lower signal quality and fewer channels but additionally provide real-time feedback on neural synchrony in naturalistic settings outside of the hyper controlled laboratory10. The tantalizing possibility of measuring memory formation effectively through these techniques led us to develop new algorithms for EEG that would satisfy these criteria.
Correlated Components Analysis (CorrCA) has emerged as a promising method for the study of inter-brain neural synchrony, enabling the extraction of maximally correlated components across individuals' neural recordings11. Projections from the hippocampus to the prefrontal region, for example, we hypothesized to show up with CorrCA while being missed by other methods. Indeed, CorrCA-derived inter-subject correlations (ISC) were found repeatedly to be related to the common cognitive aspects of experience across individuals in response to stimuli, such as engagement 11,12 interest13 and memorability3. CorrCA, as it was originally designed to utilize EEG channels data as an input, allows the computation of scalp projections, which provide valuable information about the spatial distribution of correlated neural activity across the scalp 11,14. Compared to the more familiar Canonical Correlation Analysis (CCA), CorrCA is designed to find shared components across multiple datasets or subjects while maximizing the correlation of these components across the datasets. This approach also enables multiple datasets as inputs from different subjects or experimental runs, outputting a set of components that are maximally correlated across all input datasets. A key assumption of corrCCA we accept is that there are underlying components generated by the human brain that are shared across datasets. In other words, that because the source of this data is the human brain we accept as a sufficient common element of each data source to validly map across participants by, while still allowing for individual variations in the cognitive and emotional components meaning of that data.
A benefit of methods in the corrCCA class is that we can interpret the shared components across all datasets. In this study, we present an enhanced approach to CorrCA, called features-based Correlated Components Analysis (FCCA). This method examines the synchrony level between individuals based on the dynamics of EEG activity features rather than on denoised neural activity itself, thus focusing on synchrony originating in the activity properties rather than physical localization. That is the main contribution of this paper: FCCA by orienting towards relationships in activity properties and not physical space, is able to overcome spatial limitations of EEG to extract MRI level insights for a fraction of the cost, and far more frequently since the technique is possible from data recorded at home.
Our results demonstrate the validity of FCCA, and its potential to derive insights from the dynamics of neural synchrony between subjects about engagement, emotional positivity, and memory levels. The proposed comprehensive algorithmic approach carries significance for contexts in which there is a need for robust real-time estimation of group experience in naturalistic settings, using consumer-grade recording devices and, we believe, has far reaching implications for various fields. Where memory is an important factor, such as education, entertainment, and clinical applications, FCCA can inform strategies for enhancing learning, creating memorable content, and advancing understanding of emotional engagement dynamics and memory formation in a variety of contexts.
Methods
We conducted two experiments using commercially available EEG headbands with participants who watched the same set of video clips from home. The first experiment involved short video clips viewing with subsequent emotional ratings and a memory test. The second experiment included full episode viewing followed by a memory test a few days later.
Participants
All participants had normal or corrected-to-normal vision, provided written informed consent, and received payment for their participation. Exclusion criteria included the use of medication that might influence the experiment or the presence of neurological or psychiatric conditions. Inclusion criteria required participants to have normal vision or vision corrected to normal with contact lenses. In the first experiment, there were 32 participants (mean age = 36, SD = 8.25, 16 females). In the second experiment, 27 participants (age 33, SD = 4 years, 13 females). For the memory test related to the American TV episode in experiment 2, the subjects were distributed approximately evenly across the five major regions of the continental United States (Northeast, Southwest, West, Southeast, and Midwest) and all spoke English as their native language.
Data Acquisition
In both experiments, participants viewed the videos within a mobile Android app ("Arctop” by Arctop, Inc.), while their electrical brain activity was recorded. EEG activity was recorded using Muse headbands with EEG sensors embedded. For Experiment 1, the Muse 2 headband was used, while the Muse S headband (Interaxon Inc.) was used for Experiment 2. Both devices are portable, noninvasive electroencephalography (EEG) devices with a sampling rate of 256 Hz, containing four dry EEG sensors located on the scalp at two frontal channels (AF7 and AF8) and two temporal channels positioned behind the ears (TP9 and TP10), with a reference channel at Fpz. The first experiment was conducted in an office building with a support staff available to assist the participant with the device and the experimental protocol, in the second experiment each participant was mailed an Arctop technology kit at home that included headphones (Sony), brain signal measuring headband, and a tablet computer (Samsung) with the Arctop app installed. The headbands were put on by the participants themselves, with the assistance of a Quality Assurance (QA) screen that started before each session. The QA showed the participants, in real-time, the data’ quality, easily directing them to adjust the headband properly for optimal signal quality. In both experiments, recording was performed while the participants were alone in a quiet room.
Experiment 1: Short Clips. Experiment 2: Curb Your Enthusiasm Episode.
The experiment was composed of two sessions, separated by a break. During these sessions each subject watched overall 38 short clips, each ranging from 32 to 100 seconds in length, with a median duration of 59 seconds. The video clips collection was composed of clips with positive (e.g. babies laughing), negative (e.g. An excerpt from “The Champ”15), and neutral affective valence (e.g. moving abstract shapes). In a second experimental day, adapted from the study of Hasson et al.16, participants viewed a 27-minute-long episode from the English-speaking television sitcom Curb Your Enthusiasm (Season 1, Episode 7: “AAMCO” by Larry David). The episode depicted various independent events, such as a dinner party and a minor car accident. A list of the content in table.
Emotional Ranking
In the first experiment, after watching each clip, participants ranked their emotional responses across eight emotions using a slider button in the Arctop app: enjoyment, interest, happiness, dislike, boredom, stress, relaxation, and sadness. We calculated the average score per emotion for each stimulus.
Experimental Design and Tasks.Memory test question examples for the video clips in experiment 1 including recall and recollection.Memory test question example for the long episode in experiment 2.
Memory Assessment
In both experiments, subjects weren’t aware that they would perform a memory test. Instead, after the end of the experiment they were notified that there was an additional online task that they could participate in, for which they would receive additional compensation. They received the memory test via email 3 days after the video-watching session. We verified that the email was received, and the participants were instructed to perform the test in a quiet room.
In both experiments, the test was administered via a Google Form. Each question included a still frame from the relevant time interval as a visual cue and provided three alternative forced-choice answers. Participants were instructed to choose one answer and rate their confidence level on a Likert scale ranging from 1 to 6. Memory performance was derived by calculating the percentage of correct answers per question across subjects.
For the first experiment, the memory test was composed of 3 questions about events from each of four of the clips (out of the 38 they watched), i.e. 12 questions overall. For the second experiment, the memory test consisted of 69 questions about the episode's narrative. The questions were designed so that the events they addressed were separated by approximately 20 seconds, and each question could be answered using information from the relevant segment of the episode alone.
EEG-Based Between-Subjects Synchrony Estimation
Data Preprocessing
EEG data were preprocessed to remove artifacts and filtered to retain relevant frequency bands. A band-pass filter (0.5–48 Hz) was applied to each channel together with a notch filter (either 50 Hz or 60 Hz) to remove line noise. During the performed tasks, 1.5 s of EEG data segments were extracted from the filtered signal using a sliding window with a stride of 100 ms. Data was excluded based on dropped packets of EEG data length relative to the stimulus, which occurred due to Bluetooth disconnections: deviation of more than a second was set as the threshold for exclusion in both experiments. Besides excluding data based on the recording duration threshold, recordings that had more than 5% missing values were excluded. Other data curation approaches were taken to exclude invalid data while maintaining the sample sufficient for each stimulus according to the heuristic of EEG-based CorrCA calculations are valid for eleven participants and more13. Preprocessing was completed with feature extraction, where from each EEG segment (epoch) a total of 54 features were extracted according to the current version of Arctop software platform processing (July 2024), resulting in a matrix of epochs X features X subjects created per stimulus to be used in subsequent FCCA analysis routines.
Correlated Components Analysis (CorrCA)
To examine the stimulus-evoked neural activity synchrony level between subjects, we utilized CorrCA, which was applied to yield the ISC of the neural responses 3,12,13,13,17. Correlated component analysis extracts projections of the data with maximal correlation by finding linear combinations of the EEG data or features dynamics over time. As described in detail by Cohen and Parra3, the procedure involves the calculation of the pooled between-subject cross-covariance,
and the pooled within-subject covariance,
where
measures the cross-covariance of all the time-series data sources in subject k with all sources in subject l. When CorrCA is implemented on the EEG data, these sources refer to the collected neural activity data, such as EEG-electrodes data. When calculating it on the features data, instead of a matrix of time by electrodes, the matrix is of time by features. Vector Xk(t) represents the scalp voltages, or features dynamics, at time t in subject k, and, x̄k, their mean value in time.
Obtaining the ISC involves simultaneously diagonalizing the pooled covariance and the cross-covariance of the two data sets. The linear components that achieve this can be obtained as the solutions of a generalized eigenvalue equation:
Solving for
leads to ISC per component, as the component projections that capture the largest correlation between subjects are the eigenvectors
of the matrix
with the strongest eigenvalues. Thus, the strength of correlation in the ith component is obtained by:
High ISC is obtained when the dynamics of either EEG or features are similar across subjects. Prior to computing eigenvectors, the pooled within-subject correlation matrix is regularized to improve robustness to outliers using shrinkage.
ISC Noise Level Estimation
To assess the ISC difference from noise level, i.e. remove time-dependent intersubject entrainment while maintaining each subject's temporal and spatial correlation, we implemented a phase scrambling method 18,19. This method involves selecting a random phase offset for each subject and circularly shifting all data points along the time dimension by this offset. Specifically, for each subject, we first computed a feature matrix for the full EEG length. The circular shift was then applied along the time axis of this matrix, followed by segmenting the matrix according to event labels. The circular shift was applied to each event separately and ISC was then calculated over the circularly shifted data, either over the entire event (for overall ISC) or by windows of 30 samples (for ISC dynamics). This process was repeated one hundred times, with a new circular shift applied. ISC computed in each iteration, yielding a distribution that estimated the noise for each stimulus.
ISC Significance Test
The circular-shuffle-based distribution approach allows the generation of a null distribution while maintaining the inherent structure of the original data. The ISC p-value was computed per component. The p-value of the overall ISC was calculated as 1 minus the cumulative distribution of all circular-shift ISCs for a specific component. The dynamic p-value was computed similarly for each time point.
To validate the approach of applying CorrCA on the dynamics of the feature as opposed to the EEG signals, each ISC output of both approaches was compared relative to the noise level. The p-value relative to this distribution was calculated per stimulus, and ISCs whose calculated p-value was smaller than 5% were considered significantly different from noise. We estimated the fraction of significant ISCs for the same set of stimuli, depending on the CorrCA approach.
ISC Measures
Overall ISC
For the relation of the inter-brain synchrony with the emotional ranking, we utilized the ISC for each stimulus, due to the rankings’ being referred to the entire stimulus. The ISC we refer to here is the eigenvalue of the eigenvalue problem mentioned above. We extracted this measure of the first three components.
ISC Scores
When looking into the relation of memory levels relative to the ISC level of specific events or scenes in the videos, we calculated the ISC dynamics, as it is expected different scenes evoke varying levels of ISC. We compute the correlations in a time-resolved fashion11 by employing a sliding window with a 3-s duration with a shift of the window occurring every second.
In the first experiment, we calculated the ISC scores of each of the three first components in the time windows which correspond to the parts of the video where the memory questions were asked, and their relation to the memory accuracy. The ISC dynamics were averaged within the time frame that corresponds to the event that the question refers to. The mean ISC per event was normalized by subtracting the error rate of that time frame, as calculated by the aforementioned noise estimation procedure. Note that when the average ISC in the given time window is higher than the error rate, the obtained value is positive, and vice versa.
For the episode analysis, the ISC values were also calculated in partially overlapping time windows as in the first experiment. Subjects with missing values were removed from analysis at each time window. The shuffle-based noise level per time window was subtracted from the ISC value per window, to obtain a normalized ISC curve along the episode. The normalized ISC curve within each memory test question’s relevant time frame was averaged, yielding an ISC measure per question. Both the obtained ISC and the memory performance curves underwent convolution with a moving average of 0.5 to 3 seconds running window.
Factor Analysis for Emotions Ranking
Factor analysis was performed to uncover latent structures within the emotional rankings and their relation to ISC. As a preliminary step towards conducting the factor analysis, we examined the multicollinearity among the average emotional ranking by stimulus using the Variance Inflation Factor (VIF), to ensure that the factors extracted are not influenced by redundant or highly correlated variables.
Variance Inflation Factor (VIF) Analysis
High multicollinearity, indicating redundancy, was identified for emotions receiving scores between 5 to 10. We found that enjoyment and happiness ranking exhibited moderate to high multicollinearity (VIF values of 9.45 and 5.71, respectively). To address this issue, we combined enjoyment and happiness into a single composite variable. Following this step, the revised VIF values indicate that the issue of multicollinearity has been effectively handled, with all variables displaying low to moderate multicollinearity (minimum VIF of 1.68 and Maximum VIF of 4.11) and, therefore, low redundancy.
Determining the Number of Factors
To determine the number of factors for the factor analysis, we employed the scree plot method (named for the plot's resemblance to scree rock cliffs in nature) and the Kaiser criterion (Supplementary Figure 1). Based on this analysis, we proceeded with three factors for our factor analysis. The resulting three factors in the factor analysis (after inverting their signs for clarity) can be interpreted as "Emotional Positivity," "Arousal," and "Engagement", respectively (Supplementary Figure 2). Factor 1, "Emotional Positivity," is characterized by high positive loadings for the enjoyment/happiness composite (0.581) and relaxation (0.47), and high negative loadings for stress (-0.84), dislike (-0.62), and sadness (-0.60). This factor represents a clear contrast between positive and negative emotions. Factor 2, "Arousal," includes high positive loadings for dislike (0.48) and stress (0.43), and high negative loadings for sadness (-0.74). This factor captures emotions related to reactivity, stress, and sadness, and thus can be interpreted as capturing arousal-related emotions. Notably, the second component’s ISC was found to be positively correlated with memory accuracy. Factor 3, "Engagement," is defined by high positive loadings for interest (0.77) and enjoyment/happiness composite (0.59), and high negative loadings for boredom (-0.84). This factor represents engagement versus disengagement.
These results further demonstrate the validity and efficacy of the FCCA approach in yielding robust inter-subject correlations (ISC) as the first three components revealed their distinct relationships with cognitive processes, suggesting that ISC derived from FCCA is informative about both emotion and memory processes in response to content.
Relation of ISC to Memory Scores
In Experiment 1, we investigated the relationship between memory accuracy for specific mini-scenes and ISC scores. ISC dynamics were extracted for each stimulus, and the mean ISC within the time frames corresponding to the mini-scenes was calculated and normalized by subtracting the error rate. For each of the first three components, the Shapiro-Wilk test was performed to choose between Pearson's and Spearman's correlation coefficients. The p-values were adjusted using the Bonferroni correction method. For Experiment 2, we assessed memory accuracy and ISC dynamics using a range of convolution window sizes (0.5 to 3 seconds). The correlation between memory accuracy and ISC dynamics was calculated, with the Shapiro-Wilk test performed to assess normality and choose the appropriate correlation coefficient. Corresponding p-values were adjusted for multiple comparisons using the Bonferroni correction method.
Statistical Analysis
All statistical analyses were performed via Python (version 3.12.3; “Statsmodels” and “SciPy” libraries). Pearson or Spearman correlation coefficients were chosen based on the results of the Shapiro-Wilk test for normality. The p-values that are reported refer to the hypothesis test whose null hypothesis is that two samples do not correlate. All p-values were adjusted for multiple comparisons using the Bonferroni correction method.
Results
In the current study, frontal EEG activity was recorded using a Muse headband while subjects watched video stimuli in two different designs (Muse 2 for the first design and Muse S for the second). In an offline-analysis, the data was preprocessed, EEG features were extracted and neural synchrony was calculated. In the first design, each subject watched 38 short clips (ranging from a minimum of 32 seconds to a maximum of 100 seconds, a median of 59 seconds) and was asked to rank eight emotions (enjoyment, interest, happiness, dislike, boredom, stress, relaxation, and sadness; Figure 1a) after each clip. Three days later, they performed a memory test to estimate their accuracy in recalling mini-scenes from several clips viewed during the experiment (Figure 1b). In the second design (Figure 1c), participants viewed a 27-minute-long episode and performed a memory test on mini-scenes from the episode. In both designs, the subjects watched the clips and took the memory tests alone. They were unaware that they would be taking a memory test beforehand. See the designs’ description elaboration in methods.
Features-Based CorrCA Approach Validation
We calculated ISC for each stimulus based on the EEG features. The ISC distributions of the first three components are plotted in Figure 2a. To assess their difference in the stimulus-specific noise level, we created a “circular-shuffles”-based ISC distribution (i.e. randomly shifting in time each subject’s signals to remove time-dependent intersubject entrainment). We compared the discriminability from noise for ISC values derived from performing CorrCA (i.e. calculated for the dynamics of the EEG signals) and FCCA (i.e. calculated based on the EEG-derived features (see methods for elaborated explanation). We found that the implementation of FCCA led to ISC scores that were robust and were significant relative to shuffled data (for 33 out of 38 stimuli; Figure 2b), to a greater extent relative to EEG-based CorrCA (for 1 out of 38 stimuli; Figure 2c).
FCCA Validation: distributions of ISC scores for the first three components across all stimuli.
Arctop's novel FCCA method: ISC scores derived were significantly different relative to shuffled data for 33 out of 38 stimuli, or 86.8%.Current state-of-the-art CorrCA method: ISC scores derived from current state-of-the-art CorrCA were significantly different relative to shuffled data for 1 out of 38 stimuli, or 2.6%.
Mapping Emotional Space Relative to FCCA-Derived ISCs
Having established that FCCA yields extremely robust ISCs that are approximately 30 times more informative than the nearest methods, we turned to examining the relationship between emotional responses and the ISC values obtained by our approach. Multiple emotional responses for each stimulus were ranked by each subject (see Figure 1a). Emotional responses are inherently complex and multidimensional, involving a variety of distinct but interrelated feelings (for example, feelings of happiness and enjoyment might co-occur but not be fully redundant, as might feelings of sadness and dislike).
Factor analysis enables us to reduce this complexity, and examine whether the emotional rankings share common factors that are linked to the ISC of each component separately. Due to its ability to uncover latent structures that represent the combined effects of multiple emotions, Factor analysis can potentially reveal underlying patterns that may not be apparent in pairwise comparisons. Following eliminating redundancy and determining the number of factors in a data-driven manner (see methods), factor analysis was performed for 3 factors. The resulting factors are interpreted as "Emotional Positivity," "Arousal," and "Engagement", respectively (Supplementary Figure 2).
Supplementary Figure 2: Factor-Analysis-Derived Factors Loadings For Emotional Ranking.
Visualization for the factor loadings of emotional rankings across three factors (i.e. the contribution of each emotion to the identified factors): "Emotional Positivity," "Arousal," and "Engagement." Each cell represents the loading of an emotion on a specific factor, with the color scale indicating the magnitude and direction of the loading.
Emotional Factors Correlations with ISC
To examine the relation between each of the emotional factors and the ISC of the first three components, we performed a correlation analysis and conducted Bonferroni correction for multiple comparisons. The results indicate that only the "Engagement" factor has a significant positive correlation with the ISC of the 1st component (r= 0.48, p=0.006; Figure 3a), suggesting that a higher score for this component is associated with higher engagement. "Emotional Positivity" and "Arousal" did not show significant correlations with ISC of Component 1 (r=-0.16, r=0.05; Figure 3a). The ISC of component 2 was negatively correlated with the “Emotional Positivity” factor (r=-0.44, p=0.015; Figure 3b), but not to the other two factors. Interestingly, the ISC of the third component was not found to be significantly correlated to any of the factors (Figure 3c).
The "Engagement" factor showed a significant positive correlation with ISC of the 1st component (r=0.48, **p=0.006). "Emotional Positivity" and "Arousal" factors were not significantly correlated with ISC of Component 1. The ISC of component 2 was negatively correlated with the "Emotional Positivity" factor (r=-0.44, *p=0.015). ISC of the 3rd component showed no significant correlations with any of the factors.
Figure 3: Relation of Emotional Factors with ISC Components.
Memory Decoding from FCCA-Derived ISCs
Short Clips
We investigated the relationship between the memorability of short scenes from a selection of short videos, and the ISC score (n=32, design no. 1; Figure 5a). Specifically, we examined the ISC scores of each of the three first components in the time windows corresponding to the parts of the video where the memory questions were asked, and their relation to the memory accuracy. To that end, we extracted the ISC dynamics for each stimulus, calculated the mean ISC within the time frame that corresponds to the mini-scene, and normalized it by subtracting the error rate of that time frame (see methods; example in Figure 4a). Note that when the average ISC in the given time window is higher than the error rate, the obtained value is positive, and vice versa.
A statistical analysis was performed to examine the correlation between memory accuracy and the ISC scores. After applying Bonferroni correction, we found that the only significant result was for the second component’s ISC, which had a positive Pearson correlation (r = 0.711, p = 0.0284) between memory accuracy and the ISC scores. For the first component, the analysis revealed a positive Spearman correlation (r = 0.663), however, this correlation was marginally significant (p = 0.06; Supplementary Figure 3a). For the third component, the analysis showed a weak Pearson correlation (r = 0.135, p = 0.676; Supplementary Figure 3b) between memory accuracy and the ISC scores. This correlation was not statistically significant before the Bonferroni correction.
Overall, these results suggest a specific relationship between the ISC scores of the second component and the memory accuracy for the corresponding video segments.
Figure 4: Second Component’s ISC Scores Relation to Memory Accuracy for Short Clips.
Example of ISC dynamics and the shuffle-based error rate, and the time frame corresponding to the events the questions relate to. Shows significant positive Pearson correlation between memory accuracy and ISC scores for the second component (r=0.711, **p=0.0284).
Curb Your Enthusiasm Episode
We wanted to further examine the relation between memory decoding capabilities and FCCA-generated ISC. First, we wanted to examine the reproducibility of the specific relation to the second component’s ISC that we observed in the first experiment. Second, to gain a more thorough perspective of the relation between the dynamics of these two processes, by having dozens of mini-scenes within a longer stimulus, as opposed to a diverse set where differences in brain states and latent variables of attention and emotion may encode differently due to juxtapositions in time alone. This approach allows us to characterize the relation between fluctuations in the time series pair. Lastly, since in the first experiment, the relation between the memory accuracy and the first component’s ISC was marginally significant, we sought to further examine the nature of this relation.
To that end, we adopted a previously established fMRI experimental procedure, which was designed to examine the relation between memory accuracy and the level of inter-subject synchrony, along the time course of a longer stimulus16. Specifically, we obtained memory accuracy for questions about 69 independent mini-scenes from a 35.5-minute “Curb Your Enthusiasm” episode (n=11; design no. 2) and extracted the ISC scores that corresponded to their time frames. This episode was chosen for its preponderance of independent mini-scenes or events that are disjointed from one another and thus do not necessarily contain information about one or another, which allows examining memory accuracy for specific time-points along the stimulus in a constructive way to relate it to the ISC level.
The TV show stimulus was much longer, approximately a half hour, and accordingly, the relevant events were spread over a duration of several seconds. We accounted for the possible shift in time between the ISC and the memory level time series in a data-driven manner by performing a correlation analysis of the ISC score during the mini-scenes timing to the accuracy in answering these questions, while smoothing both time series with a moving average. Specifically, the correlation between the memory level and the ISC score dynamics was calculated for a range of running average convolution window sizes ranging from 0.5 seconds to 3 seconds, in increments of 0.5 seconds. A significance test was performed to each convolution value and the corresponding p-values were adjusted for multiple comparisons using the Bonferroni correction method.
We found that the relation between the memory accuracy and the second component’s ISC score dynamic was replicated in this design. The correlation in the first convolution window size of 0.5 seconds was found to be positive, however it was marginally significant, (r= 0.45, p=0.09). After applying the Bonferroni correction, we found that all other comparisons were statistically significant (for 1 second p<0.05, for 1.5 and above, p<0.0001). This result further corroborates the relation of the 2nd component’s ISC and emphasizes its stability. Figure 5a depicts the surprisingly stable and robust relation between the ISC and memory accuracy of the 2.5-second convolution.
Figure 5: Memory Accuracy and ISC Dynamics in the “Curb Your Enthusiasm” Episode. Scatter plot of the ISC vs. memory accuracy as measured by multiple choice quiz answers showing that ISC score derived from FCCA is predictive of future episodic memory performance.
After corroborating the relation of memory accuracy to the 2nd component’s ISC, we turned to examine its relation to the 1st component’s ISC, as it was marginally significant in the first experiment. We conducted the same set of analyses and statistical tests and found no significant, or marginally significant results (Figure 5c). This finding strengthens the specificity of the memory accuracy relation of the FCCA-derived ISC of the second component.
Discussion
Our study introduces a novel approach to CorrCA, termed FCCA, which examines synchrony based on EEG activity features. We found this method to be highly robust, compatible with consumer-grade EEG devices, and allows obtaining insights into the common emotional and cognitive experiences based on the neural synchrony between subjects.
After establishing that on our data set, FCCA yields results that differ from noise at a higher rate than CorrCA, we found that the FCCA approach revealed distinct relationships between ISC of the first two components and the video-clips’ emotional ranking by the subjects. Our analysis showed that the "Engagement" emotional factor strongly associated with the first component’s ISC. This result aligns with previous research on the relationship between neural synchrony and engagement 3,19,20, and therefore corroborates the validity of the FCCA approach.
Importantly, our study extends beyond engagement to explore the relationship between neural synchrony, as measured by FCCA, and memory which is known to be largely housed anotomically in deep structures that are closed-fields to EEG signal. We found that the relation between ISC and memory was specific to the second component. This result was evident both when examining the memory for selected events taken from several short video clips and also when these events were taken from the time span of a longer episode. Our findings are in agreement with the previous findings of Cohen and Parra3, who demonstrated the relation between the first 3 components’ ISC relation to memory level when applying the CorrCA analysis to EEG data. Therefore, the ISC dynamics derived by our algorithm can effectively capture memorability level dynamics while individuals experience the same stimulus.
The second component’s overall ISC and the "Emotional Positivity" factor were negatively correlated. This finding aligns with the study of Nummenmaa et al.21, reporting an association between negative valence and increased inter-brain synchrony in the emotion-processing network, including prefrontal regions. Together with our result that the second component’s ISC level corresponds to the memorability dynamics along the stimulus, our findings point to an inverse relationship between emotional positivity and memory accuracy. This finding is in agreement with evidence suggesting that sad or negative stimuli are more memorable6. Future research could explore how the dynamics of emotional valence along the stimulus correspond to the ISC dynamics, and how different emotional valences in context interact with memory formation.
As FCCA provides a comprehensive framework that is suitable for either offline or real-time tracking of emotional and cognitive processes simultaneously, future studies could also examine how different types of stimuli, ranging from educational materials to various forms of entertainment, affect experience. This approach may facilitate creation of personalized content in education and entertainment, potentially informing the design of engaging and memorable content. Finally, future studies could extend the use of this robust algorithm to the hyperscanning research field, by enabling tracking emotions, and memory processes based on large groups neural dynamics.
Previous findings that relate the synchrony level calculated by CorrCA to individual traits, such as age22 and familiarity or expertise with respect to the displayed visual stimuli23 (Goldberg, 2014). Therefore, given the potential applicability of FCCA for naturalistic settings, it may be implemented for identifying divergence from a common synchronized neural response, allowing the detection of abnormal patterns in individuals’ traits, such as emotional reactivity or cognitive processing. This avenue may be highly relevant in educational and clinical settings, in the diagnosis and treatment of disorders affecting social cognition or memory.
In conclusion, our study demonstrates that our proposed approach is a powerful and versatile tool for investigating neural synchrony. By providing a nuanced understanding of how emotional responses and memory processes are reflected in neural synchrony, FCCA opens new pathways for studying common patterns within and between collective experiences across various contexts. The ability of FCCA to yield meaningful results across different kinds of consumer-grade recording devices significantly enhances its generalizability, expanding the scope of real-time tracking of neural synchrony in naturalistic settings that is possible through a variety of headwear from earbuds with temporally located EEG sensors to AR/VR devices with frontally located EEG sensors.
Supplementary
Eigenvalues of the principal components in descending order for emotional factors. The Kaiser criterion, which retains factors with eigenvalues greater than 1, supported retaining three factors.
Supplementary Figure 1: Scree plot for choosing the number of factors for Factor Analysis.
References
1. Nummenmaa, L., Lahnakoski, J. M. & Glerean, E. Sharing the social world via intersubject neural synchronization. Curr. Opin. Psychol. 24, 7–14 (2018).
2. Hasson, U. & Frith, C. D. Mirroring and beyond: coupled dynamics as a generalized framework for modeling social interactions. Philos. Trans. R. Soc. B Biol. Sci. 371, 20150366 (2016).
3. Cohen, S. S. & Parra, L. C. Memorable Audiovisual Narratives Synchronize Sensory and Supramodal Neural Responses. eNeuro 3, (2016).
4. Kensinger, E. A. Remembering the Details: Effects of Emotion. Emot. Rev. 1, 99–113 (2009).
5. Santamaria, L. et al. Emotional valence modulates the topology of the parent-infant inter-brain network. NeuroImage 207, 116341 (2020).
6. Cunningham, T. J., Leal, S. L., Yassa, M. A. & Payne, J. D. Post-encoding stress enhances mnemonic discrimination of negative stimuli. Learn. Mem. 25, 611–619 (2018).
7. McGaugh, J. L. The amygdala modulates the consolidation of memories of emotionally arousing experiences. Annu. Rev. Neurosci. 27, 1–28 (2004).
8. Czeszumski, A. et al. Hyperscanning: A Valid Method to Study Neural Inter-brain Underpinnings of Social Interaction. Front. Hum. Neurosci. 14, (2020).
9. Dumas, G., Nadel, J., Soussignan, R., Martinerie, J. & Garnero, L. Inter-Brain Synchronization during Social Interaction. PLoS ONE 5, e12166 (2010).
10. Nazneen, T. et al. Recent Trends in Non-invasive Neural Recording Based Brain-to-Brain Synchrony Analysis on Multidisciplinary Human Interactions for Understanding Brain Dynamics: A Systematic Review. Front. Comput. Neurosci. 16, 875282 (2022).
11. Dmochowski, J. P., Sajda, P., Dias, J. & Parra, L. C. Correlated Components of Ongoing EEG Point to Emotionally Laden Attention – A Possible Marker of Engagement? Front. Hum. Neurosci. 6, (2012).
12. Poulsen, A. T., Kamronn, S., Dmochowski, J., Parra, L. C. & Hansen, L. K. EEG in the classroom: Synchronized neural recordings during video presentation. Sci. Rep. 7, 43916 (2017).
13. Dmochowski, J. P. et al. Audience preferences are predicted by temporal reliability of neural processing. Nat. Commun. 5, 4567 (2014).
14. Truong, N. C. D. et al. Mapping and understanding of correlated electroencephalogram (EEG) responses to the newsvendor problem. Sci. Rep. 12, 13800 (2022).
15. Gross, J. J. & Levenson, R. W. Emotion elicitation using films. Cogn. Emot. 9, 87–108 (1995).
16. Hasson, U., Furman, O., Clark, D., Dudai, Y. & Davachi, L. Enhanced Intersubject Correlations during Movie Viewing Correlate with Successful Episodic Encoding. Neuron 57, 452–462 (2008).
17. Cohen, S. S. et al. Neural engagement with online educational videos predicts learning performance for individual students. Neurobiol. Learn. Mem. 155, 60–64 (2018).
18. Prichard, D. & Theiler, J. Generating surrogate data for time series with several simultaneously measured variables. Phys. Rev. Lett. 73, 951–954 (1994).
19. Parra, L. C., Haufe, S. & Dmochowski, J. P. Correlated Components Analysis - Extracting Reliable Dimensions in Multivariate Data. Neurons Behav. Data Anal. Theory 2, (2019).
20. Dikker, S. et al. Brain-to-Brain Synchrony Tracks Real-World Dynamic Group Interactions in the Classroom. Curr. Biol. 27, 1375–1380 (2017).
21. Nummenmaa, L. et al. Emotions promote social interaction by synchronizing brain activity across individuals. Proc. Natl. Acad. Sci. 109, 9599–9604 (2012).
22. Petroni, A. et al. The Variability of Neural Responses to Naturalistic Videos Change with Age and Sex. eNeuro 5, ENEURO.0244-17.2017 (2018).
23. Madsen, J., Margulis, E. H., Simchy-Gross, R. & Parra, L. C. Music synchronizes brainwaves across listeners with strong effects of repetition, familiarity and training. Sci. Rep. 9, 3576 (2019).
24. Goldberg, H., Preminger, S., & Malach, R. (2014). The emotion-action link? Naturalistic emotional stimuli preferentially activate the human dorsal visual stream. Neuroimage, 84, 254-264.
25. Lee, H., Chen, J. Predicting memory from the network structure of naturalistic events. Nat Commun 13, 4235 (2022).
26. Rugg, M. D. & Curran, T. Event-related potentials and recognition memory. Trends Cogn. Sci. 11, 251–257 (2007).
27. Rugg, M. D. & Vilberg, K. L. Brain networks underlying episodic memory retrieval. Curr. Opin. Neurobiol. 23, 255–260 (2013).
28. Simony, E. et al. Dynamic reconfiguration of the default mode network during narrative comprehension. Nat. Commun. 7, 12141 (2016).
29. Goldberg, I. I., Harel, M. & Malach, R. When the brain loses its self: prefrontal inactivation during sensorimotor processing. Neuron 50, 329–339 (2006).
30. Summerfield, C. et al. Neocortical connectivity during episodic memory formation. PLoS Biol. 4, e128 (2006).