Spring 2022 Alliyah's Figures1) Abstract
Despite it being one of the visual cortex’s most important functions, little is known about how we process familiarity (how much experience one has with visual stimuli). Past studies have hypothesized that, when presented with images with higher familiarity, select areas in the visual cortex would increase average blood-oxygen-level-dependent (BOLD) signals. However, no study has directly examined familiarity’s effect on the visual cortex neurobehaviorally. Here we asked participants to rank images of urban and rural scenes by their familiarity to the participants’ everyday lives. We used a linear regression to confirm our hypothesis that individuals would rank images corresponding to their own living environment as more familiar. We then delved into the BOLD5000 fMRI database and examined the average BOLD signal in the visual cortex for the highest- and lowest-ranked images. We also examined if there was a stronger BOLD signal overall within images considered urban since most of the participants were from that living environment. The data supported the fact that there was a bias between living environments and image rankings—participants were more likely to rank images corresponding to their living environments as more familiar. Familiarity thus appears to have a significant effect on the workings of the visual cortex.
2) Introduction
Imagine you are driving along the road at night when a wayward vehicle suddenly swerves towards you. Luckily, your reflexes kick in, and you are able to turn away and avoid an accident in the knick of time.
Our brain has an amazing ability to process visual data within 13 milliseconds [14]. This may be attributed to how much of our operating system is dedicated to the sense of sight: thirty percent of our brain’s resources are allocated to vision, with only a meager 8-10% allocated towards all other senses [13]. The visual cortex is the region in the occipital lobe most responsible for the processing of visual information, and it interfaces with the temporal and parietal lobes to facilitate visual processing and recognition.
Within the occipital lobe, visual information is processed hierarchically through the six centers of the visual cortex (V1-V6). There are several factors that impact how an image is processed. Specifically for scenes, some of the low-level, less abstracted features include color, edges, and spatial frequency (light and dark distribution); the higher level image quality measures include distance, expanse, and familiarity. Familiarity is defined as how often a scene or object is encountered in everyday life. It is widely believed that familiarity has a profound effect on the modulation of certain cortical areas in the brain during visual recognition. Familiarity itself relates to strength of connection and speed of recognition. If a concept is viewed more and more often, it becomes familiar. These concept cell networks are more interconnected and often exhibit higher BOLD signals than the periphery [1][9-10].
After processing, the impulses are relayed to the temporal lobe ventral stream. The ventral stream adds color to images as well as references them with past memories (elements of image categorization and recognition). The temporal lobe is able to do this by using concept cells, neurons that specifically respond to a certain item or place called a “concept” (e.g. Dog). Concept cells with similar meanings (e.g. Dog and Cat) are generally located closer together and may be networked. It is widely accepted that the more concept cell networks are used, the more quickly we can visually recognize the concept. These networks have familiarity, causing cells to be more interconnected and greater speed of recognition [21]. Specifically, in the hippocampus, a region that facilitates long-term memory, (medial temporal lobe - MTL), there are areas of concept cells that network extensively with the visual cortex. For example, the parahippocampal place area (PPA) and retrosplenial complex (RSC) are sensitive to the viewing of scenes/places meaning they are scene-selective. Outside the temporal lobe, the lateral occipital cortex (LOC) is highly sensitive to the processing of objects (object-selective), and the occipital place area (OPA) is scene-selective. The scene-selective regions of interest (ROIs) for this study are the PPA, RSC, and OPA. Another region of interest is the EVC (early visual cortex), an area of the brain responsible for low level image processing [12]. This will be this study’s control area since it is unlikely to be modulated by high-level familiarity.
Functional magnetic resonance imaging (fMRI) is often used to analyze the visual cortex due to its great spatial resolution and ability to image sub-regional (e.g. lobe) activities using blood-oxygen-level-dependent (BOLD) signals. Regions of the brain receiving more oxygenated blood must be working harder than regions with higher levels of deoxygenated blood, so high BOLD signals correspond to higher brain activity. Thus, by comparing BOLD signals, we can rate regional activity [12].
2.1) Past Studies
An associative processing study using fMRI to compare how subjects processed isolated components of images (e.g. isolated objects or isolated scenes from images) versus complete images (e.g. images with both objects and scenes) suggested that, contrary to previous thought, processing greatly relied on connections between concept cells and connections between overarching brain regions. Given the importance of interconnected concept cells in processing, this further suggests that visual processing is more holistic than previously thought, such that familiarity may also play a role in visual processing.
Another study [9] reviewed several studies addressing scene processing and highlighted the need for studies to ascertain which variable high-level and low-level features are most paramount in determining how scenes are processed. The study also noted that multi-faceted activations deriving from a visual stimulus could be independently triggered or caused by each other (similar to [2]’s interpretation) and that researchers should aim to better differentiate between these two conditions. The study finally hinted that image detection and categorization may not occur sequentially but simultaneously. This study failed however to look into familiarity.
Furthermore, [10] found that certain areas of MTL were triggered by flashback events and images with higher scaled familiarity measures. The study also found the amygdala was face-sensitive and the hippocampus was receptive to recollection. This study showed the importance of familiarity within image processing. This study had too low a sample size though.
The BOLD5000 database [1] consists of matrices of BOLD signals experienced by 3 participants while viewing over 5,000 images. These BOLD signals were from the left and right LOC, EVC, OPA, PPA, and RSC respectively. The over 5,000 unique images were derived from three sources: Scene understandings (SUN, just scenes) [22], ImageNet (objects) [23], and Common Objects in Context (COCO, scenes and objects) [23]. Both the cognitive data and images from this dataset will be utilized by the current study.
However, no previous study has directly assessed the variable of scene familiarity on visual recognition both neurologically and behaviorally. Additionally, none have done so with a significant sample size. Past studies have also neglected to examine the effects of this variable on selective areas in both the occipital lobe and ventral visual stream (PPA, RSC, OPA, etc.).
2.2) Goals and Hypothesis
The goals for this study are to:
3) Methods
200 scenes were chosen from the BOLD5000 [1] SUN (Scene Understandings) [22] database; half of them were labeled as rural and the other half urban. The label suburban was avoided so that we could directly label images binarily as either urban or suburban and avoid confusion. However, we did include participants describing their environment as suburban in this study (in order to examine how they ranked images compared to rural and urban individuals).
We administered an online survey asking participants to rank images on a 1-5 scale, with 1 being not familiar at all and 5 being extremely familiar. Participants, before answering questions, were provided a definition of familiarity [10] (Familiarity is defined as how normal something is to you or how much experience you have with it.) and a consent form with instructions to limit confusion. All questions were mandatory. Participants received a maximum time of 1 hour to complete the 200 questions (About 18 seconds per question). [7] showed that participants, despite only quickly seeing small fractions of images, could already classify objects, validating our study’s choice to provide such a large volume of images. To avoid bias, participants were asked in a multiple choice format whether their living area was urban, rural, or suburban only after all the image questions. This is so they would have no knowledge of the hidden variable: living environment.
3.2) Survey Participants
N = 259 adults from the United States were acquired using Amazon’s Mechanical TURK and were awarded a small monetary stipend for completing the survey. All data was de-identified and questions at the survey’s beginning affirmed that participants knew the study’s purpose, how their data was protected, and that they could stop at any time or choose not to participate.
3.3) Google Sheets Analysis
Using a binary scale, rural images were given a value 0 and urban images were given a 1. Similarly rural participants were 0, and urban 1. Suburban participants were valued at 0.5. The data was analyzed using a linear regression to see whether there was an overall correlation between individuals living areas and their familiarity scoring for images fitting that category. This was done to examine if there was a bias (represented as higher familiarity score) towards either category as a result of living environment. The 3 images rated most familiar and the 3 images rated least familiar were also computed overall for all of the participants by comparing the questions’ average ratings. A t-test was then run comparing the top and bottom three images to the average familiarity ranking of other scenes not classified at the extremes (i.e. those that were not most or least familiar). This was done to prove that these images were not only at the extremes but that they were significantly above the average. Image resolutions, blurriness, and size were made uniform to eliminate extra variables that may have affected visual processing. Also, any images that may have evoked an emotional response were removed by [1]. In our analysis, any participant that answered less than 95% of the questions was rejected.
3.4) SPSS Analysis
To ensure the validity of our analysis we directly viewed interactions between the 3 living environment groups in this study through added statistical measures. Individuals were divided into three categories, rural environment, suburban environment, and urban environment. Next, a repeated measure was run ANOVA looking at the effect of the environment groups on picture scores. This analysis normalized the within-subject variance, and was multivariate. The program not only computed p-values but significance power values, allowing us to further justify the strength of our data.
3.5) BOLD5000 participants
The sample size for the BOLD5000 study was N = 4 adults from the Pittsburgh, Pennsylvania area. However, our study only utilized the first 3 since the 4th participant did not complete image viewing. Although BOLD5000 only had 4 participants, each participant viewed a large volume of images (5,254). Our study has a larger sample (N= 259) to measure the familiarity variable.
3.6) Most and Least Familiar images ROI analysis
The most and least familiar images found by the survey had their corresponding fMRI neural data from the BOLD 5000 [1] experiment examined. BOLD signal values were stored in matrices. Each row corresponds to a different image from the previous study [1]. The images were scrambled for each participant and therefore each participant had unique matrix rows for each image. Therefore, for the top 3 and bottom 3 images we delved into the raw data for each participant and extracted the matrix numbers for each (use the “find” function in Excel). Then, using MatLab, we averaged the right and left hemisphere BOLD signal values for each participant and combined them into one list. The experimental ROIs for this analysis were the PPA, OPA, RSC, and LOC. The PPA, OPA, RSC, were chosen as experimental ROIs since they are known to be modulated by scene-content [1][2][6][9]. The LOC was included since it is sensitive to objects, which are often found embedded in scenes. All the experimental ROIs were high-level areas and would therefore likely be affected by familiarity. The EVC was chosen as a control area since it would likely not be affected by the variable familiarity, as it is only a low-level feature processing area. Then, a bar graph was made for each participant showing the average differences in BOLD signal for unfamiliar versus familiar images in each ROI [18]. Then averaged graphs were averaged to view the overall trend between the 3 participants. After inspecting for visual trends, multiple paired t-tests were run in Google Sheets using the XL Miner plugin between all the familiar and unfamiliar right and left hemisphere values of each ROI. The tests were paired because we were looking at the same voxels each time. This was done to ensure that we were not just relying on visual trends, but that there was also a numerical significance between familiar and unfamiliar induced BOLD signals.
3.7) Rural and Urban Images ROI analysis.
We also sought to discern if there was a significant difference in average BOLD signal in the specific ROIs (PPA, OPA, RSC, EVC) for rural versus urban images. We extracted matrix numbers using our own programs (Appendix). Notably, before running any of these functions the column raw data was transposed to a row using an online tool to make it take up less line space and memory within the javascript. Once these matrix numbers were found, we averaged left and right hemisphere BOLD signal values for each ROI (PPA, OPA, RSC, EVC) in response to rural and urban images (Appendix). Individual bar charts were generated for each participant and then averaged across all subjects to see the general trends. T-Tests were computed within each ROI to see if there was a tangible difference between rural and urban induced BOLD signal. This step will show whether environmental familiarity affects BOLD signals neurologically.
Crucially, the raw fMRI data had already been preprocessed using normalization filters, regressing out of nuisance features (Heart Rate, respiration, etc.), motion correction and several other measures [1].
The BOLD5000 open source data is available online at: https://bold5000.github.io/. In our study, all analysis and coding was done using the programs Matlab, Google Sheets XL Miner Plugin, SPSS, and an online JavaScript compiler.
4) Results (Figures Can Be Found Here: Link to Figures)
4.1) Behavioral Data
Figure 1. T-Test between familiar average picture scores and the mean of the average familiarity scores of images not labeled as most or least familiar. An unpaired t-test assuming unequal variances was run on the averages of the 3 computed most familiar images and the average of the remaining images. Interestingly there was also significance two-tailed represented by a number < 0.001***.
Figure 2. T-Test between unfamiliar average picture scores and the mean of the average familiarity scores of images not labeled as most or least familiar. An unpaired t-test assuming unequal variances was run on the averages of the 3 computed least familiar images and the average of the remaining images (Those not located at the extremes therefore not categorized as most or least familiar). The relationship was not significant two-tailed.
Figure 3. Linear Regression between living environment and urban picture familiarity score.
A Linear regression was run with the discrete 0-1 living environment scaled responses (0 = rural, 1= urban, 0.5 = suburban) of all the participants as the input, and the resulting average urban picture score of each survey-taker as the output. There was a correlation coefficient of 0.0321 and a linear relationship represented by the equation y= 0.3781x+3.0519. Within this equal y represents the familiarity score and x represents the living environment. The p value <.001. When inputting Urban (1) one receives the number 3.43 in the equation. For suburban (0.5) it is 3.241 and rural 3.0519.
4.2.2) SPSF Analysis
Table 1: Multivariate Pairwise T-Test examining effect of urban environment on picture scores. This figure shows that the urban environment group does have a significant effect on the two types of picture scores. The power value for this relationship is exceptionally strong.
Table 2: Multivariate Pairwise T-Test examining effect of suburban environment on picture scores. There was a significant interaction between the suburban picture group and the overall picture score of the two categories. The power value is somewhat strong.
Table 3: Repeated Measures Anova run to examine if there was a bais between picture score (urban or rural) and subject group. This test ran general statistics on all data and compared all permutations. This figure shows that urbanPicScore was significantly correlated with the urban and suburban environment groups (p<0.01**) and rural (p<0.05*). RuralPicScore was only significantly correlated with the suburban picture group (p<0.05*) . However, the power values for these significances are fairly average.
4.2) Neurological Data
Figure 4: Average BOLD Signal between familiar and unfamiliar images of 3 participants in Left and Right ROIs. Visually it can be seen that the EVC, OPA, PPA, RSC, and RHLOC all favor familiar. While only the LHLOC favors unfamiliar.
Figure 5: Unpaired T-test between RSC right hemisphere unfamiliar BOLD signal and RSC right hemisphere familiar BOLD signal. The RHRSC significantly favors familiar images.
Figure 6: Unpaired T-test between RSC left hemisphere unfamiliar BOLD signal and RSC left hemisphere familiar BOLD signal. The LHRSC favors familiar images.
Figure 7: Average BOLD signal in ROIs for 3 participants between urban and rural labeled images. It can be seen visually that the EVC overall favors urban images, so does the OPA, PPA, and RSC. However in the LOC, the Left hemisphere favors rural while the right favors urban.
Figure 8: Linear regression between familiarity ranking of unfamiliar and familiar images and corresponding average BOLD signal in the RHRSC.This figure shows that there is a direct and somewhat strong linear relationship (Represented by R-Value) between BOLD signal of familiar and unfamiliar images and their familiarity scores within the RHRSC. The P-value for this relationship was 0.05 (marginal). The equation for this line is y= -0.058+0.0044x. In this relationship x represents the familiarity score while y represents the BOLD signal yielded in the RHRSC. Notably, a non-significant relationship was found in the LHRSC.
5.1) Experimental Objectives
Our goals in this experiment were:
1)To examine the effect of the variable familiarity on the average BOLD signal within certain ROIs during the viewing of BOLD5000 scene images.
2) To ascertain if living environment drives familiarity.
3) To verify selective experimental ROIs (PPA. RSC, OPA, LOC) as seen in past studies [1][2][6][9] continuously exhibit significantly higher average BOLD signals than the EVC (Control area) .
5.2) Behavioral Data Discussion
We found that behavioral data supported the latter part of our first hypothesis. The 3 most familiar images computed by the survey were: (listed in order from least to most familiar) airplaneCabin5, rooftop4, and airplanecabin6. The 3 least familiar images were: ( listed in order from least to most unfamiliar) deck5, cave4, and cornfield2. Within the study (see Figures 1 and 2), it was proven that the average rating of the least three familiar images were significantly higher than the averages of images not rated at the extremes*(P value < 0.05*). Similarly, the three most familiar images were also significantly different from images not rated as unfamiliar or familiar (p value <0.001).
It seems participants had a more extreme reaction to the familiar images than the unfamiliar ones, although both interactions were statistically significant. Perhaps this occurred due to the concept cell functioning described in[12]. Neuronal connections become stronger and more interconnected and when an idea is more familiar (It is reinforced throughout daily life). Study [21] establishes this by showing that greater activity in neuronal circuits increases cellular myelination (Fatty coating that increases transmission speed), and therefore interconnection and speed of recall. The networks for unfamiliar images would be far weaker than those for familiar images perhaps justifying the more extreme reaction to familiarity.
The data also partially supports the latter part of our third hypothesis. There was a significant relationship between living environment and average urban picture score (See Figure 3 and Table 4). Table 1 also showed that being a member of the urban experimental group had a significant effect on overall picture score. Table 2 showed this same relationship but for suburban individuals. Table 4 showed that the suburban picture had significant effects on rural picture score (Not seen in the urban picture group) and urban picture score. However, the same could not be said for rural picture score. This relationship was far from being significant with a p value of 0.49. Also, within table 4, rural pic did not have a significant effect on picture score (Table 4).
There are several reasons this may have occurred. First, rural environments characteristically have more open expanses and less explicit items to classify compared to the cluttering in urban scenes. Due to their lack of objects, rural images would not modulate the LOC as much, therefore lowering brain activity and as a result and possibly familiarity score. Further corroboration of this idea is that the top unfamiliar images were all rural and all the top familiar images were all urban.
According to the equation listed in Figure 3, urban inhabitants are significantly biased towards rating urban images higher.While rural inhabitants tend to rate urban images the lowest but not significantly so. Suburban inhabitants rate images between these two extremes. This is unsurprising given that suburban is defined as being a mixture of rural and urban features and supports our 3rd hypothesized claim about the suburban group. Table 4 once again also proves that suburban individuals are unbiased since they had a significant interaction with both rural and urban picture scores. However, that number was more significant for urban. It almost seems that given the data the term urban is becoming somewhat synonymous with familiar.
The data corroborates our hypothesis that for urban individuals would rate urban images higher, suburbaners would rank in the middle and rural lowest. It also supports that suburban individuals are generally unbiased between rural and urban images. The data in addition seems to argue that rural individuals have a harder time differentiating between rural and urban pictures than suburban and urban inhabitants. This is a notion most likely caused by their limited exposure to such environments.
5.3) Neurological Data Discussion
The Neurological data supports hypothesis 1 but only in the RSC. Figure 5 shows that in the RHRSC that familiar images yield significantly higher BOLD signals. Figure 6 shows that same interaction but for the LHRSC. Figure 8 shows that only in the RHRSC does familiarity score linearly affect the BOLD signals within the RSC (With a strong correlation, P-value is marginal). This relationship is direct, meaning that higher familiarity scores should cause higher BOLD signals. According to figure 4, more than half of the ROIs reacted higher to familiar images in at least half of their cortex (LOC and PPA reacted this way laterally, while OPA and RSC did bilaterally).
This relationship may have been localized to the RSC due to its unique role within the visual cortex. [17] performed an experiment extracting the networks of ROIs within the visual cortex. It found that a posterior network consisting of the OPA and anterior PPA while an anterior network contained the RSC and posterior PPA. This study states that the anterior network seemed to be more involved in navigation and memory. Considering that familiarity is a factor crucial to the facilitation of navigation, it is logical RSC would be significantly affected by this factor (Figures 5-8).
Our results also seem to be partially consistent with the neurological studies [18] and [20]. which found images of familiar locations caused higher BOLD signals in the PPA, RSC, and the Transverse Occipital Sulcus. However, we did not observe the same effect in the PPA as in the RSC by this principle. This may be due to a trend seen in [17] in that the PPA is a member of both circuits (anterior and posterior) and therefore had a marginal effect towards familiarity. Both of these studies are in agreement with [17] regarding the role of the RSC in navigational memory. More research may be necessary to effectively label which cortical areas are specifically triggered by familiarity.
The LOC’s object-selective tendencies justify its bias towards familiar images. As established before, all the top 3 familiar images were urban and contained more objects. It was unexpected however, that the RSC was only significantly correlated to familiarity in the right hemisphere (Figure 8). According to study [19], the right hemisphere is more involved in deductive reasoning, a process often used in visual recognition. When something is more familiar, deduction can become simpler, perhaps justifying why the right hemisphere would have a greater reaction towards this facilitating variable (familiarity). Unexpectedly, the EVC, our supposed control area, visually had a preference for unfamiliar (Figure 4). We believe this would not occur since the EVC is a low-level area and would likely be unaffected by familiarity. It is important to realize that all the unfamiliar images were rural and lacked explicit objects. However, many of the unfamiliar rural images like cornfield3 had a patterned look. Just examining the photo ,it is possible to hypothesize that a computer could easily be able to reconstruct the other half of the image when given only one part of it. However, for familiar urban images like rooftop4 with so much variability this task would prove more difficult. The EVC is known for handling low-level features such as this pattern-recognition, so if there is a higher pattern in unfamiliar it would logically lean towards this category. Further research may be necessary to evaluate if the EVC is sensitive to familiarity. To test if BOLD signal was correlated to familiarity score, we also extracted the BOLD signals of the top unfamiliar and familiar images in Matlab. Wen then ran a linear regression on BOLD signals of each ROI bilaterally compared to the familiarity score value of the image (figure 8). It was interesting that this relationship was only significant in the right hemisphere of the RSC (The other hemisphere has a p value of 0.69). Perhaps, one hemisphere responds temporally sooner to the other or there is another circuit it works with (Similarly to how the PPA is a part of both an anterior and posterior circuit [17]).
Since our control area showed some modulation, we also visually checked if our Experimental ROIs still had comparatively higher BOLD than our control area (Figures 4 and 7). In all cortical areas except the LOC this rang true. This only partially supports hypothesis 2. The LOC may also have experienced a lower BOLD signal since once again participants were viewing scenes not objects and although there may have been some embedded objects in the scenes, they were not paramount.
Hypothesis 3 was not supported neurologically. Participants from this study did not experience significantly higher BOLD signals for urban images in any ROI (Figure 7). However, visually as seen in figure 7, 4 out 5 ROIs experience an overall higher BOLD signal in reaction to urban images. This relationship was closest to being significant in the RSC (P value of 0.09). Perhaps there are variables that may affect familiarity not examined in this experiment (e.g. specific city or town structure, amount of time living in environment etc.).
5.4) Limitations and Future Research
Although this experiment achieved all of its objectives there were several limitations. First, we could not affix all 259 participants with fMRI headsets to examine responses. Our neurological analysis was limited to only 3 participants in the BOLD 5000 fMRI database. Also, the BOLD5000 study used 5,254 images but for the purpose of making the survey shorter only 200 scene images (From the SUN database) were used [1]. Although the results received were still significant it is always beneficial to test if results remain the same when a stimuli number is increased.
Real life scenes are not static like the images used in this study, so variables such as motion and timing handled by the VC’s dorsal stream were not incorporated [12]. A future research proposition would be to repeat the same study but with Graphic interchange format images (GIFS), short video clips, or virtual reality to examine the effect while under fMRI or EEG imaging. A study by [15] documents the use of such technologies for military training, an possible application of our study (Examine if training soldiers in environments similar to future combat zones is beneficial).
Consequently, fMRI itself is not very portable or accessible (Expensive). Therefore it is harder to apply principles found in this study to real-world applications such as in the classroom or driving academies. Imaging methods (e.g. Electroencephalogram and Magnetoencephalogram) used in [8] would be better prospects. We could even apply these technologies to pictorial step-tied teaching (Used in [16]) to examine if more familiar images increase brain activity and student attention within the classroom.
We also did not include suburban images in this study which may have changed the results. We felt it was unrealistic to include such images since there is a “gray area” in deciding whether a picture has equal amounts of rural and urban elements.
Finally, it is more difficult to find rural participants than urban and suburban ones. This is consistent with the well-known fact that population tends to concentrate itself in urban areas while it’s more spread out and possibly lesser in rural areas. According to [11], only 1 and 5 Americans live in a rural environment, constituting only 19.3% of the United States population. Within this study out of 259 participants, 70 were rural (27%), 97 were urban (37%), and 92 were suburban (36%). Although the margin between suburban and urban is fairly small (1%) there is a sizable difference in the rural percentage (10%) which may have affected our results. This data may also be used to test the hypothesis that driving in an unfamiliar territory will raise the chances of getting into an accident by hindering visual cortex functioning.
Future work may examine how our research relates to nostalgia (a sentimental longing for things of the past), a concept similar to familiarity, with the addition of more emotional response. Perhaps nostalgia would target more of the basal ganglia and amygdala (part of the brain’s limbic system-processes sensation and determines emotional responses [12]). It would be necessary to compare the neurological differences between visual nostalgia and familiarity to corroborate this notion. In our study specifically, it was found that participants reacted exponentially more to the top three familiar images statistically (P value <0.001 ***) than the top three unfamiliar ones (P value of 0.05*). It would be compelling to see if this relationship also lends itself to concepts described as nostalgic. This could then be applied to advertising since by using images that are more familiar and perhaps more nostalgic companies may be able to elicit a higher neurological response in viewers [3].
6) Conclusion
In conclusion, we performed this research aimed to help bridge the gap in understanding of the visual cortex by exploring a new variable: Familiarity. This variable had never been examined neurologically and behaviorally within selective ROIs of the visual cortex. We created and analyzed a survey to examine the effects of living environment on familiarity and to extract extreme images. We then applied these results to neurological analysis and examined BOLD signals in response to different image calibers. We found the data supported all hypotheses neurobehavioral at least partially. Our research can be applied in the fields of driving, teaching, military, and overall in understanding the human brain.
About the Author
Alliyah Steele is a freshman at Harvard College intending to concentrate in CMBB and Astrophysics with a secondary in Applied Mathematics.
References
1. Chang, N., Pyles, J. A., Marcus, A., Gupta, A., Tarr, M. J., & Aminoff, E. M. (2019). Bold5000, a public fmri dataset while viewing 5000 visual images. Scientific data, 6(1):49.
2. Aminoff, E. M., & Tarr, M. J. (2015). Associative processing is inherent in scene perception. PLoS One 10:e0128840. 10.1371/journal.pone.0128840
3. Oliver, M. B., & Bartsch, A. (2010). Appreciation as Audience Response: Exploring Entertainment Gratifications Beyond Hedonism, Human Communication Research, 36(1), 53–81, https://doi.org/10.1111/j.1468-2958.2009.01368.x
4. Neural systems for visual orienting and their relationships to spatial working memory J. Cogn. Neurosci., 14 (2002), pp. 508-523
5. Yarkoni, T., Barch, D. M., Gray, J. R., Conturo, T. E., & Braver, T. S. (2009). BOLD correlates of trial-by-trial reaction time variability in gray and white matter: A multi-study fMRI analysis. PLoS One, 4, e4257.
6. Aminoff, E. M., Kveraga, K. & Bar, M. (2013). The role of the parahippocampal cortex in cognition. Trends Cogn. Sci. 17, 379–390
7. Grill-Spector, K. & Kanwisher, N. (2005). Visual recognition: as soon as you know it is there, you know what it is. Psychol. Sci. 16, 152–160
8. Liu, H., Agam, Y., Madsen, J. R., & Kreiman, G. (2009). Timing, timing, timing: Fast decoding of object information from intracranial field potentials in human visual cortex. Neuron, 62, 281–290
9. Malcolm, G. L., Groen, I. I. A., & Baker, C .I. (2016). Making sense of real-world scenes. Trends Cogn. Sci. 20, 843–856.
10. Kafkas, A., Migo, E. M., Morris, R. G., Kopelman, M. D., Montaldi, D., & Mayes, A. R. Material specificity drives medial temporal lobe familiarity but not hippocampal recollection.
11. Ratcliffe, M., Burd, C., Holder, K. & Fields, A. (2016). Defining rural at the US census bureau American Community Survey and Geography Brief (Washington, DC). United States Census Bureau.
12. Brain Facts: a Primer on the Brain and Nervous System. Society for Neuroscience, 2012.
13. Grady, D. (1993, June 1). The eye and brain work in a partnership to interpret conflicting signals from the outside world. Ultimately, we see whatever our brains think we should. Retrieved from http://discovermagazine.com/1993/jun/thevisionthingma227.
14. Trafton, A. (2014, January 16). In the blink of an eye. Retrieved from http://news.mit.edu/2014/in-the-blink-of-an-eye-0116.
15. Jun, G. & Xiong, Y. Virtual Reality and Its Application on Military Surveying, Mapping and Operating Simulation. Journal Of The Pla Institute Of Surveying And Mapping, 1996-02.
16. Sadoski, M. (2006, Sept 2). “A Dual Coding View of Vocabulary Learning.” Reading & Writing Quarterly, 2, 221–238. doi:10.1080/10573560590949359.
17. Baldassano, C., Esteva, A., Fei-Fei, L., & Beck, D. M. (2016). Two distinct scene processing networks 971 connecting vision and memory. eNeuro. 10.1523:1–14.
18. Epstein R. A., Higgins, J. S., Jablonski, K., & Feiler, A. M. (2007). Visual scene processing in familiar and unfamiliar environments, J Neurophysiol, 97, 3670-3683.
19. Parsons, L. M. & Osherson, D. (2001). New Evidence for Distinct Right and Left Brain Systems for Deductive versus Probabilistic Reasoning. Cerebral Cortex, 11(10), 954–965. https://doi.org/10.1093/cercor/11.10.954
20. Epstein, R. A., Parker, W. E., & Feiler, A. M. (2007).Where am I now? Distinct roles for parahippocampal and retrosplenial cortices in place recognition, J Neurosci, 27, 6141-6149.
21. Hasan, M., Suresh Kanna, M., Jun, W., Surendran Ramkrishnan, A., Iqbar, Z., Lee, Y., & Li, Y. (2019). Schema‐like learning and memory consolidation acting through myelination. FASAB Journal, 33(11).
22. Xiao, J., Ehinger, K. A., Hays, J., Torralba, A., & Oliva, A. (2014, Aug 13). “SUN Database: Exploring a Large Collection of Scene Categories.” International Journal of Computer Vision, 119(1). https://doi.org/10.1007/s11263-014-0748-y
23. Fei-Fei, L., Deng, J., & Li, K. (2010). ImageNet: Constructing a large-scale image database. Journal of Vision, 9(8), 1037–1037. https://doi.org/10.1167/9.8.1037
24. Lin, T.-Y., Maire, M., Belongie, S., Bourdev, L., Girshick, R., Hays, J., Perona, P., Ramanan, D., Zitnick, C. L., & Dolla, P. (2014, May 1). “Microsoft COCO: Common Objects in Context.” Cornell University. arxiv.org/pdf/1405.0312.pdf.
Despite it being one of the visual cortex’s most important functions, little is known about how we process familiarity (how much experience one has with visual stimuli). Past studies have hypothesized that, when presented with images with higher familiarity, select areas in the visual cortex would increase average blood-oxygen-level-dependent (BOLD) signals. However, no study has directly examined familiarity’s effect on the visual cortex neurobehaviorally. Here we asked participants to rank images of urban and rural scenes by their familiarity to the participants’ everyday lives. We used a linear regression to confirm our hypothesis that individuals would rank images corresponding to their own living environment as more familiar. We then delved into the BOLD5000 fMRI database and examined the average BOLD signal in the visual cortex for the highest- and lowest-ranked images. We also examined if there was a stronger BOLD signal overall within images considered urban since most of the participants were from that living environment. The data supported the fact that there was a bias between living environments and image rankings—participants were more likely to rank images corresponding to their living environments as more familiar. Familiarity thus appears to have a significant effect on the workings of the visual cortex.
2) Introduction
Imagine you are driving along the road at night when a wayward vehicle suddenly swerves towards you. Luckily, your reflexes kick in, and you are able to turn away and avoid an accident in the knick of time.
Our brain has an amazing ability to process visual data within 13 milliseconds [14]. This may be attributed to how much of our operating system is dedicated to the sense of sight: thirty percent of our brain’s resources are allocated to vision, with only a meager 8-10% allocated towards all other senses [13]. The visual cortex is the region in the occipital lobe most responsible for the processing of visual information, and it interfaces with the temporal and parietal lobes to facilitate visual processing and recognition.
Within the occipital lobe, visual information is processed hierarchically through the six centers of the visual cortex (V1-V6). There are several factors that impact how an image is processed. Specifically for scenes, some of the low-level, less abstracted features include color, edges, and spatial frequency (light and dark distribution); the higher level image quality measures include distance, expanse, and familiarity. Familiarity is defined as how often a scene or object is encountered in everyday life. It is widely believed that familiarity has a profound effect on the modulation of certain cortical areas in the brain during visual recognition. Familiarity itself relates to strength of connection and speed of recognition. If a concept is viewed more and more often, it becomes familiar. These concept cell networks are more interconnected and often exhibit higher BOLD signals than the periphery [1][9-10].
After processing, the impulses are relayed to the temporal lobe ventral stream. The ventral stream adds color to images as well as references them with past memories (elements of image categorization and recognition). The temporal lobe is able to do this by using concept cells, neurons that specifically respond to a certain item or place called a “concept” (e.g. Dog). Concept cells with similar meanings (e.g. Dog and Cat) are generally located closer together and may be networked. It is widely accepted that the more concept cell networks are used, the more quickly we can visually recognize the concept. These networks have familiarity, causing cells to be more interconnected and greater speed of recognition [21]. Specifically, in the hippocampus, a region that facilitates long-term memory, (medial temporal lobe - MTL), there are areas of concept cells that network extensively with the visual cortex. For example, the parahippocampal place area (PPA) and retrosplenial complex (RSC) are sensitive to the viewing of scenes/places meaning they are scene-selective. Outside the temporal lobe, the lateral occipital cortex (LOC) is highly sensitive to the processing of objects (object-selective), and the occipital place area (OPA) is scene-selective. The scene-selective regions of interest (ROIs) for this study are the PPA, RSC, and OPA. Another region of interest is the EVC (early visual cortex), an area of the brain responsible for low level image processing [12]. This will be this study’s control area since it is unlikely to be modulated by high-level familiarity.
Functional magnetic resonance imaging (fMRI) is often used to analyze the visual cortex due to its great spatial resolution and ability to image sub-regional (e.g. lobe) activities using blood-oxygen-level-dependent (BOLD) signals. Regions of the brain receiving more oxygenated blood must be working harder than regions with higher levels of deoxygenated blood, so high BOLD signals correspond to higher brain activity. Thus, by comparing BOLD signals, we can rate regional activity [12].
2.1) Past Studies
An associative processing study using fMRI to compare how subjects processed isolated components of images (e.g. isolated objects or isolated scenes from images) versus complete images (e.g. images with both objects and scenes) suggested that, contrary to previous thought, processing greatly relied on connections between concept cells and connections between overarching brain regions. Given the importance of interconnected concept cells in processing, this further suggests that visual processing is more holistic than previously thought, such that familiarity may also play a role in visual processing.
Another study [9] reviewed several studies addressing scene processing and highlighted the need for studies to ascertain which variable high-level and low-level features are most paramount in determining how scenes are processed. The study also noted that multi-faceted activations deriving from a visual stimulus could be independently triggered or caused by each other (similar to [2]’s interpretation) and that researchers should aim to better differentiate between these two conditions. The study finally hinted that image detection and categorization may not occur sequentially but simultaneously. This study failed however to look into familiarity.
Furthermore, [10] found that certain areas of MTL were triggered by flashback events and images with higher scaled familiarity measures. The study also found the amygdala was face-sensitive and the hippocampus was receptive to recollection. This study showed the importance of familiarity within image processing. This study had too low a sample size though.
The BOLD5000 database [1] consists of matrices of BOLD signals experienced by 3 participants while viewing over 5,000 images. These BOLD signals were from the left and right LOC, EVC, OPA, PPA, and RSC respectively. The over 5,000 unique images were derived from three sources: Scene understandings (SUN, just scenes) [22], ImageNet (objects) [23], and Common Objects in Context (COCO, scenes and objects) [23]. Both the cognitive data and images from this dataset will be utilized by the current study.
However, no previous study has directly assessed the variable of scene familiarity on visual recognition both neurologically and behaviorally. Additionally, none have done so with a significant sample size. Past studies have also neglected to examine the effects of this variable on selective areas in both the occipital lobe and ventral visual stream (PPA, RSC, OPA, etc.).
2.2) Goals and Hypothesis
The goals for this study are to:
- Examine the effect of familiarity on average BOLD signals during scene viewing.
- Examine the relationship between participants’ living environments and their perceived familiarity of images of various living environments.
- Verify the specificity of scene-selective regions of the brain (PPA, RSC, OPA, LOC, EVC) to scene stimuli [1][2][6][9].
- Individuals will exhibit higher BOLD signals for familiar images compared to unfamiliar ones. The top three unfamiliar and familiar images will also be rated significantly different from the average.
- The RSC, PPA, LOC, and OPA will have higher average BOLD signals than the control ROI the EVC [1][2][6][9]. This will occur since familiarity is a high-level feature and therefore would only affect high-level cortical areas.
- The participants in the BOLD5000 study will have significantly higher average BOLD signal for urban scenes compared to rural ones since they live in an urban environment (Pittsburgh) [1]. Also, within the survey urban participants will rate urban images higher while rural participants will rate rural higher. Suburban participants will show no bias to either category.
3) Methods
200 scenes were chosen from the BOLD5000 [1] SUN (Scene Understandings) [22] database; half of them were labeled as rural and the other half urban. The label suburban was avoided so that we could directly label images binarily as either urban or suburban and avoid confusion. However, we did include participants describing their environment as suburban in this study (in order to examine how they ranked images compared to rural and urban individuals).
We administered an online survey asking participants to rank images on a 1-5 scale, with 1 being not familiar at all and 5 being extremely familiar. Participants, before answering questions, were provided a definition of familiarity [10] (Familiarity is defined as how normal something is to you or how much experience you have with it.) and a consent form with instructions to limit confusion. All questions were mandatory. Participants received a maximum time of 1 hour to complete the 200 questions (About 18 seconds per question). [7] showed that participants, despite only quickly seeing small fractions of images, could already classify objects, validating our study’s choice to provide such a large volume of images. To avoid bias, participants were asked in a multiple choice format whether their living area was urban, rural, or suburban only after all the image questions. This is so they would have no knowledge of the hidden variable: living environment.
3.2) Survey Participants
N = 259 adults from the United States were acquired using Amazon’s Mechanical TURK and were awarded a small monetary stipend for completing the survey. All data was de-identified and questions at the survey’s beginning affirmed that participants knew the study’s purpose, how their data was protected, and that they could stop at any time or choose not to participate.
3.3) Google Sheets Analysis
Using a binary scale, rural images were given a value 0 and urban images were given a 1. Similarly rural participants were 0, and urban 1. Suburban participants were valued at 0.5. The data was analyzed using a linear regression to see whether there was an overall correlation between individuals living areas and their familiarity scoring for images fitting that category. This was done to examine if there was a bias (represented as higher familiarity score) towards either category as a result of living environment. The 3 images rated most familiar and the 3 images rated least familiar were also computed overall for all of the participants by comparing the questions’ average ratings. A t-test was then run comparing the top and bottom three images to the average familiarity ranking of other scenes not classified at the extremes (i.e. those that were not most or least familiar). This was done to prove that these images were not only at the extremes but that they were significantly above the average. Image resolutions, blurriness, and size were made uniform to eliminate extra variables that may have affected visual processing. Also, any images that may have evoked an emotional response were removed by [1]. In our analysis, any participant that answered less than 95% of the questions was rejected.
3.4) SPSS Analysis
To ensure the validity of our analysis we directly viewed interactions between the 3 living environment groups in this study through added statistical measures. Individuals were divided into three categories, rural environment, suburban environment, and urban environment. Next, a repeated measure was run ANOVA looking at the effect of the environment groups on picture scores. This analysis normalized the within-subject variance, and was multivariate. The program not only computed p-values but significance power values, allowing us to further justify the strength of our data.
3.5) BOLD5000 participants
The sample size for the BOLD5000 study was N = 4 adults from the Pittsburgh, Pennsylvania area. However, our study only utilized the first 3 since the 4th participant did not complete image viewing. Although BOLD5000 only had 4 participants, each participant viewed a large volume of images (5,254). Our study has a larger sample (N= 259) to measure the familiarity variable.
3.6) Most and Least Familiar images ROI analysis
The most and least familiar images found by the survey had their corresponding fMRI neural data from the BOLD 5000 [1] experiment examined. BOLD signal values were stored in matrices. Each row corresponds to a different image from the previous study [1]. The images were scrambled for each participant and therefore each participant had unique matrix rows for each image. Therefore, for the top 3 and bottom 3 images we delved into the raw data for each participant and extracted the matrix numbers for each (use the “find” function in Excel). Then, using MatLab, we averaged the right and left hemisphere BOLD signal values for each participant and combined them into one list. The experimental ROIs for this analysis were the PPA, OPA, RSC, and LOC. The PPA, OPA, RSC, were chosen as experimental ROIs since they are known to be modulated by scene-content [1][2][6][9]. The LOC was included since it is sensitive to objects, which are often found embedded in scenes. All the experimental ROIs were high-level areas and would therefore likely be affected by familiarity. The EVC was chosen as a control area since it would likely not be affected by the variable familiarity, as it is only a low-level feature processing area. Then, a bar graph was made for each participant showing the average differences in BOLD signal for unfamiliar versus familiar images in each ROI [18]. Then averaged graphs were averaged to view the overall trend between the 3 participants. After inspecting for visual trends, multiple paired t-tests were run in Google Sheets using the XL Miner plugin between all the familiar and unfamiliar right and left hemisphere values of each ROI. The tests were paired because we were looking at the same voxels each time. This was done to ensure that we were not just relying on visual trends, but that there was also a numerical significance between familiar and unfamiliar induced BOLD signals.
3.7) Rural and Urban Images ROI analysis.
We also sought to discern if there was a significant difference in average BOLD signal in the specific ROIs (PPA, OPA, RSC, EVC) for rural versus urban images. We extracted matrix numbers using our own programs (Appendix). Notably, before running any of these functions the column raw data was transposed to a row using an online tool to make it take up less line space and memory within the javascript. Once these matrix numbers were found, we averaged left and right hemisphere BOLD signal values for each ROI (PPA, OPA, RSC, EVC) in response to rural and urban images (Appendix). Individual bar charts were generated for each participant and then averaged across all subjects to see the general trends. T-Tests were computed within each ROI to see if there was a tangible difference between rural and urban induced BOLD signal. This step will show whether environmental familiarity affects BOLD signals neurologically.
Crucially, the raw fMRI data had already been preprocessed using normalization filters, regressing out of nuisance features (Heart Rate, respiration, etc.), motion correction and several other measures [1].
The BOLD5000 open source data is available online at: https://bold5000.github.io/. In our study, all analysis and coding was done using the programs Matlab, Google Sheets XL Miner Plugin, SPSS, and an online JavaScript compiler.
4) Results (Figures Can Be Found Here: Link to Figures)
4.1) Behavioral Data
Figure 1. T-Test between familiar average picture scores and the mean of the average familiarity scores of images not labeled as most or least familiar. An unpaired t-test assuming unequal variances was run on the averages of the 3 computed most familiar images and the average of the remaining images. Interestingly there was also significance two-tailed represented by a number < 0.001***.
Figure 2. T-Test between unfamiliar average picture scores and the mean of the average familiarity scores of images not labeled as most or least familiar. An unpaired t-test assuming unequal variances was run on the averages of the 3 computed least familiar images and the average of the remaining images (Those not located at the extremes therefore not categorized as most or least familiar). The relationship was not significant two-tailed.
Figure 3. Linear Regression between living environment and urban picture familiarity score.
A Linear regression was run with the discrete 0-1 living environment scaled responses (0 = rural, 1= urban, 0.5 = suburban) of all the participants as the input, and the resulting average urban picture score of each survey-taker as the output. There was a correlation coefficient of 0.0321 and a linear relationship represented by the equation y= 0.3781x+3.0519. Within this equal y represents the familiarity score and x represents the living environment. The p value <.001. When inputting Urban (1) one receives the number 3.43 in the equation. For suburban (0.5) it is 3.241 and rural 3.0519.
4.2.2) SPSF Analysis
Table 1: Multivariate Pairwise T-Test examining effect of urban environment on picture scores. This figure shows that the urban environment group does have a significant effect on the two types of picture scores. The power value for this relationship is exceptionally strong.
Table 2: Multivariate Pairwise T-Test examining effect of suburban environment on picture scores. There was a significant interaction between the suburban picture group and the overall picture score of the two categories. The power value is somewhat strong.
Table 3: Repeated Measures Anova run to examine if there was a bais between picture score (urban or rural) and subject group. This test ran general statistics on all data and compared all permutations. This figure shows that urbanPicScore was significantly correlated with the urban and suburban environment groups (p<0.01**) and rural (p<0.05*). RuralPicScore was only significantly correlated with the suburban picture group (p<0.05*) . However, the power values for these significances are fairly average.
4.2) Neurological Data
Figure 4: Average BOLD Signal between familiar and unfamiliar images of 3 participants in Left and Right ROIs. Visually it can be seen that the EVC, OPA, PPA, RSC, and RHLOC all favor familiar. While only the LHLOC favors unfamiliar.
Figure 5: Unpaired T-test between RSC right hemisphere unfamiliar BOLD signal and RSC right hemisphere familiar BOLD signal. The RHRSC significantly favors familiar images.
Figure 6: Unpaired T-test between RSC left hemisphere unfamiliar BOLD signal and RSC left hemisphere familiar BOLD signal. The LHRSC favors familiar images.
Figure 7: Average BOLD signal in ROIs for 3 participants between urban and rural labeled images. It can be seen visually that the EVC overall favors urban images, so does the OPA, PPA, and RSC. However in the LOC, the Left hemisphere favors rural while the right favors urban.
Figure 8: Linear regression between familiarity ranking of unfamiliar and familiar images and corresponding average BOLD signal in the RHRSC.This figure shows that there is a direct and somewhat strong linear relationship (Represented by R-Value) between BOLD signal of familiar and unfamiliar images and their familiarity scores within the RHRSC. The P-value for this relationship was 0.05 (marginal). The equation for this line is y= -0.058+0.0044x. In this relationship x represents the familiarity score while y represents the BOLD signal yielded in the RHRSC. Notably, a non-significant relationship was found in the LHRSC.
5.1) Experimental Objectives
Our goals in this experiment were:
1)To examine the effect of the variable familiarity on the average BOLD signal within certain ROIs during the viewing of BOLD5000 scene images.
2) To ascertain if living environment drives familiarity.
3) To verify selective experimental ROIs (PPA. RSC, OPA, LOC) as seen in past studies [1][2][6][9] continuously exhibit significantly higher average BOLD signals than the EVC (Control area) .
5.2) Behavioral Data Discussion
We found that behavioral data supported the latter part of our first hypothesis. The 3 most familiar images computed by the survey were: (listed in order from least to most familiar) airplaneCabin5, rooftop4, and airplanecabin6. The 3 least familiar images were: ( listed in order from least to most unfamiliar) deck5, cave4, and cornfield2. Within the study (see Figures 1 and 2), it was proven that the average rating of the least three familiar images were significantly higher than the averages of images not rated at the extremes*(P value < 0.05*). Similarly, the three most familiar images were also significantly different from images not rated as unfamiliar or familiar (p value <0.001).
It seems participants had a more extreme reaction to the familiar images than the unfamiliar ones, although both interactions were statistically significant. Perhaps this occurred due to the concept cell functioning described in[12]. Neuronal connections become stronger and more interconnected and when an idea is more familiar (It is reinforced throughout daily life). Study [21] establishes this by showing that greater activity in neuronal circuits increases cellular myelination (Fatty coating that increases transmission speed), and therefore interconnection and speed of recall. The networks for unfamiliar images would be far weaker than those for familiar images perhaps justifying the more extreme reaction to familiarity.
The data also partially supports the latter part of our third hypothesis. There was a significant relationship between living environment and average urban picture score (See Figure 3 and Table 4). Table 1 also showed that being a member of the urban experimental group had a significant effect on overall picture score. Table 2 showed this same relationship but for suburban individuals. Table 4 showed that the suburban picture had significant effects on rural picture score (Not seen in the urban picture group) and urban picture score. However, the same could not be said for rural picture score. This relationship was far from being significant with a p value of 0.49. Also, within table 4, rural pic did not have a significant effect on picture score (Table 4).
There are several reasons this may have occurred. First, rural environments characteristically have more open expanses and less explicit items to classify compared to the cluttering in urban scenes. Due to their lack of objects, rural images would not modulate the LOC as much, therefore lowering brain activity and as a result and possibly familiarity score. Further corroboration of this idea is that the top unfamiliar images were all rural and all the top familiar images were all urban.
According to the equation listed in Figure 3, urban inhabitants are significantly biased towards rating urban images higher.While rural inhabitants tend to rate urban images the lowest but not significantly so. Suburban inhabitants rate images between these two extremes. This is unsurprising given that suburban is defined as being a mixture of rural and urban features and supports our 3rd hypothesized claim about the suburban group. Table 4 once again also proves that suburban individuals are unbiased since they had a significant interaction with both rural and urban picture scores. However, that number was more significant for urban. It almost seems that given the data the term urban is becoming somewhat synonymous with familiar.
The data corroborates our hypothesis that for urban individuals would rate urban images higher, suburbaners would rank in the middle and rural lowest. It also supports that suburban individuals are generally unbiased between rural and urban images. The data in addition seems to argue that rural individuals have a harder time differentiating between rural and urban pictures than suburban and urban inhabitants. This is a notion most likely caused by their limited exposure to such environments.
5.3) Neurological Data Discussion
The Neurological data supports hypothesis 1 but only in the RSC. Figure 5 shows that in the RHRSC that familiar images yield significantly higher BOLD signals. Figure 6 shows that same interaction but for the LHRSC. Figure 8 shows that only in the RHRSC does familiarity score linearly affect the BOLD signals within the RSC (With a strong correlation, P-value is marginal). This relationship is direct, meaning that higher familiarity scores should cause higher BOLD signals. According to figure 4, more than half of the ROIs reacted higher to familiar images in at least half of their cortex (LOC and PPA reacted this way laterally, while OPA and RSC did bilaterally).
This relationship may have been localized to the RSC due to its unique role within the visual cortex. [17] performed an experiment extracting the networks of ROIs within the visual cortex. It found that a posterior network consisting of the OPA and anterior PPA while an anterior network contained the RSC and posterior PPA. This study states that the anterior network seemed to be more involved in navigation and memory. Considering that familiarity is a factor crucial to the facilitation of navigation, it is logical RSC would be significantly affected by this factor (Figures 5-8).
Our results also seem to be partially consistent with the neurological studies [18] and [20]. which found images of familiar locations caused higher BOLD signals in the PPA, RSC, and the Transverse Occipital Sulcus. However, we did not observe the same effect in the PPA as in the RSC by this principle. This may be due to a trend seen in [17] in that the PPA is a member of both circuits (anterior and posterior) and therefore had a marginal effect towards familiarity. Both of these studies are in agreement with [17] regarding the role of the RSC in navigational memory. More research may be necessary to effectively label which cortical areas are specifically triggered by familiarity.
The LOC’s object-selective tendencies justify its bias towards familiar images. As established before, all the top 3 familiar images were urban and contained more objects. It was unexpected however, that the RSC was only significantly correlated to familiarity in the right hemisphere (Figure 8). According to study [19], the right hemisphere is more involved in deductive reasoning, a process often used in visual recognition. When something is more familiar, deduction can become simpler, perhaps justifying why the right hemisphere would have a greater reaction towards this facilitating variable (familiarity). Unexpectedly, the EVC, our supposed control area, visually had a preference for unfamiliar (Figure 4). We believe this would not occur since the EVC is a low-level area and would likely be unaffected by familiarity. It is important to realize that all the unfamiliar images were rural and lacked explicit objects. However, many of the unfamiliar rural images like cornfield3 had a patterned look. Just examining the photo ,it is possible to hypothesize that a computer could easily be able to reconstruct the other half of the image when given only one part of it. However, for familiar urban images like rooftop4 with so much variability this task would prove more difficult. The EVC is known for handling low-level features such as this pattern-recognition, so if there is a higher pattern in unfamiliar it would logically lean towards this category. Further research may be necessary to evaluate if the EVC is sensitive to familiarity. To test if BOLD signal was correlated to familiarity score, we also extracted the BOLD signals of the top unfamiliar and familiar images in Matlab. Wen then ran a linear regression on BOLD signals of each ROI bilaterally compared to the familiarity score value of the image (figure 8). It was interesting that this relationship was only significant in the right hemisphere of the RSC (The other hemisphere has a p value of 0.69). Perhaps, one hemisphere responds temporally sooner to the other or there is another circuit it works with (Similarly to how the PPA is a part of both an anterior and posterior circuit [17]).
Since our control area showed some modulation, we also visually checked if our Experimental ROIs still had comparatively higher BOLD than our control area (Figures 4 and 7). In all cortical areas except the LOC this rang true. This only partially supports hypothesis 2. The LOC may also have experienced a lower BOLD signal since once again participants were viewing scenes not objects and although there may have been some embedded objects in the scenes, they were not paramount.
Hypothesis 3 was not supported neurologically. Participants from this study did not experience significantly higher BOLD signals for urban images in any ROI (Figure 7). However, visually as seen in figure 7, 4 out 5 ROIs experience an overall higher BOLD signal in reaction to urban images. This relationship was closest to being significant in the RSC (P value of 0.09). Perhaps there are variables that may affect familiarity not examined in this experiment (e.g. specific city or town structure, amount of time living in environment etc.).
5.4) Limitations and Future Research
Although this experiment achieved all of its objectives there were several limitations. First, we could not affix all 259 participants with fMRI headsets to examine responses. Our neurological analysis was limited to only 3 participants in the BOLD 5000 fMRI database. Also, the BOLD5000 study used 5,254 images but for the purpose of making the survey shorter only 200 scene images (From the SUN database) were used [1]. Although the results received were still significant it is always beneficial to test if results remain the same when a stimuli number is increased.
Real life scenes are not static like the images used in this study, so variables such as motion and timing handled by the VC’s dorsal stream were not incorporated [12]. A future research proposition would be to repeat the same study but with Graphic interchange format images (GIFS), short video clips, or virtual reality to examine the effect while under fMRI or EEG imaging. A study by [15] documents the use of such technologies for military training, an possible application of our study (Examine if training soldiers in environments similar to future combat zones is beneficial).
Consequently, fMRI itself is not very portable or accessible (Expensive). Therefore it is harder to apply principles found in this study to real-world applications such as in the classroom or driving academies. Imaging methods (e.g. Electroencephalogram and Magnetoencephalogram) used in [8] would be better prospects. We could even apply these technologies to pictorial step-tied teaching (Used in [16]) to examine if more familiar images increase brain activity and student attention within the classroom.
We also did not include suburban images in this study which may have changed the results. We felt it was unrealistic to include such images since there is a “gray area” in deciding whether a picture has equal amounts of rural and urban elements.
Finally, it is more difficult to find rural participants than urban and suburban ones. This is consistent with the well-known fact that population tends to concentrate itself in urban areas while it’s more spread out and possibly lesser in rural areas. According to [11], only 1 and 5 Americans live in a rural environment, constituting only 19.3% of the United States population. Within this study out of 259 participants, 70 were rural (27%), 97 were urban (37%), and 92 were suburban (36%). Although the margin between suburban and urban is fairly small (1%) there is a sizable difference in the rural percentage (10%) which may have affected our results. This data may also be used to test the hypothesis that driving in an unfamiliar territory will raise the chances of getting into an accident by hindering visual cortex functioning.
Future work may examine how our research relates to nostalgia (a sentimental longing for things of the past), a concept similar to familiarity, with the addition of more emotional response. Perhaps nostalgia would target more of the basal ganglia and amygdala (part of the brain’s limbic system-processes sensation and determines emotional responses [12]). It would be necessary to compare the neurological differences between visual nostalgia and familiarity to corroborate this notion. In our study specifically, it was found that participants reacted exponentially more to the top three familiar images statistically (P value <0.001 ***) than the top three unfamiliar ones (P value of 0.05*). It would be compelling to see if this relationship also lends itself to concepts described as nostalgic. This could then be applied to advertising since by using images that are more familiar and perhaps more nostalgic companies may be able to elicit a higher neurological response in viewers [3].
6) Conclusion
In conclusion, we performed this research aimed to help bridge the gap in understanding of the visual cortex by exploring a new variable: Familiarity. This variable had never been examined neurologically and behaviorally within selective ROIs of the visual cortex. We created and analyzed a survey to examine the effects of living environment on familiarity and to extract extreme images. We then applied these results to neurological analysis and examined BOLD signals in response to different image calibers. We found the data supported all hypotheses neurobehavioral at least partially. Our research can be applied in the fields of driving, teaching, military, and overall in understanding the human brain.
About the Author
Alliyah Steele is a freshman at Harvard College intending to concentrate in CMBB and Astrophysics with a secondary in Applied Mathematics.
References
1. Chang, N., Pyles, J. A., Marcus, A., Gupta, A., Tarr, M. J., & Aminoff, E. M. (2019). Bold5000, a public fmri dataset while viewing 5000 visual images. Scientific data, 6(1):49.
2. Aminoff, E. M., & Tarr, M. J. (2015). Associative processing is inherent in scene perception. PLoS One 10:e0128840. 10.1371/journal.pone.0128840
3. Oliver, M. B., & Bartsch, A. (2010). Appreciation as Audience Response: Exploring Entertainment Gratifications Beyond Hedonism, Human Communication Research, 36(1), 53–81, https://doi.org/10.1111/j.1468-2958.2009.01368.x
4. Neural systems for visual orienting and their relationships to spatial working memory J. Cogn. Neurosci., 14 (2002), pp. 508-523
5. Yarkoni, T., Barch, D. M., Gray, J. R., Conturo, T. E., & Braver, T. S. (2009). BOLD correlates of trial-by-trial reaction time variability in gray and white matter: A multi-study fMRI analysis. PLoS One, 4, e4257.
6. Aminoff, E. M., Kveraga, K. & Bar, M. (2013). The role of the parahippocampal cortex in cognition. Trends Cogn. Sci. 17, 379–390
7. Grill-Spector, K. & Kanwisher, N. (2005). Visual recognition: as soon as you know it is there, you know what it is. Psychol. Sci. 16, 152–160
8. Liu, H., Agam, Y., Madsen, J. R., & Kreiman, G. (2009). Timing, timing, timing: Fast decoding of object information from intracranial field potentials in human visual cortex. Neuron, 62, 281–290
9. Malcolm, G. L., Groen, I. I. A., & Baker, C .I. (2016). Making sense of real-world scenes. Trends Cogn. Sci. 20, 843–856.
10. Kafkas, A., Migo, E. M., Morris, R. G., Kopelman, M. D., Montaldi, D., & Mayes, A. R. Material specificity drives medial temporal lobe familiarity but not hippocampal recollection.
11. Ratcliffe, M., Burd, C., Holder, K. & Fields, A. (2016). Defining rural at the US census bureau American Community Survey and Geography Brief (Washington, DC). United States Census Bureau.
12. Brain Facts: a Primer on the Brain and Nervous System. Society for Neuroscience, 2012.
13. Grady, D. (1993, June 1). The eye and brain work in a partnership to interpret conflicting signals from the outside world. Ultimately, we see whatever our brains think we should. Retrieved from http://discovermagazine.com/1993/jun/thevisionthingma227.
14. Trafton, A. (2014, January 16). In the blink of an eye. Retrieved from http://news.mit.edu/2014/in-the-blink-of-an-eye-0116.
15. Jun, G. & Xiong, Y. Virtual Reality and Its Application on Military Surveying, Mapping and Operating Simulation. Journal Of The Pla Institute Of Surveying And Mapping, 1996-02.
16. Sadoski, M. (2006, Sept 2). “A Dual Coding View of Vocabulary Learning.” Reading & Writing Quarterly, 2, 221–238. doi:10.1080/10573560590949359.
17. Baldassano, C., Esteva, A., Fei-Fei, L., & Beck, D. M. (2016). Two distinct scene processing networks 971 connecting vision and memory. eNeuro. 10.1523:1–14.
18. Epstein R. A., Higgins, J. S., Jablonski, K., & Feiler, A. M. (2007). Visual scene processing in familiar and unfamiliar environments, J Neurophysiol, 97, 3670-3683.
19. Parsons, L. M. & Osherson, D. (2001). New Evidence for Distinct Right and Left Brain Systems for Deductive versus Probabilistic Reasoning. Cerebral Cortex, 11(10), 954–965. https://doi.org/10.1093/cercor/11.10.954
20. Epstein, R. A., Parker, W. E., & Feiler, A. M. (2007).Where am I now? Distinct roles for parahippocampal and retrosplenial cortices in place recognition, J Neurosci, 27, 6141-6149.
21. Hasan, M., Suresh Kanna, M., Jun, W., Surendran Ramkrishnan, A., Iqbar, Z., Lee, Y., & Li, Y. (2019). Schema‐like learning and memory consolidation acting through myelination. FASAB Journal, 33(11).
22. Xiao, J., Ehinger, K. A., Hays, J., Torralba, A., & Oliva, A. (2014, Aug 13). “SUN Database: Exploring a Large Collection of Scene Categories.” International Journal of Computer Vision, 119(1). https://doi.org/10.1007/s11263-014-0748-y
23. Fei-Fei, L., Deng, J., & Li, K. (2010). ImageNet: Constructing a large-scale image database. Journal of Vision, 9(8), 1037–1037. https://doi.org/10.1167/9.8.1037
24. Lin, T.-Y., Maire, M., Belongie, S., Bourdev, L., Girshick, R., Hays, J., Perona, P., Ramanan, D., Zitnick, C. L., & Dolla, P. (2014, May 1). “Microsoft COCO: Common Objects in Context.” Cornell University. arxiv.org/pdf/1405.0312.pdf.
