Research Area: Cognitive Neuroscience

​perception, memory, language, social cognition, attention, empathy, brain disorders, aging, cognitive flexibility, developing brain

Can we think without language?

by Anna Ivanova |

As part of our Ask the Brain series, Anna Ivanova, a graduate student who studies how the brain processes language in the labs of Nancy Kanwisher and Evelina Fedorenko, answers the question, “Can we think without language?”

Anna Ivanova headshot
Graduate student Anna Ivanova studies language processing in the brain.

_____

Imagine a woman – let’s call her Sue. One day Sue gets a stroke that destroys large areas of brain tissue within her left hemisphere. As a result, she develops a condition known as global aphasia, meaning she can no longer produce or understand phrases and sentences. The question is: to what extent are Sue’s thinking abilities preserved?

Many writers and philosophers have drawn a strong connection between language and thought. Oscar Wilde called language “the parent, and not the child, of thought.” Ludwig Wittgenstein claimed that “the limits of my language mean the limits of my world.” And Bertrand Russell stated that the role of language is “to make possible thoughts which could not exist without it.” Given this view, Sue should have irreparable damage to her cognitive abilities when she loses access to language. Do neuroscientists agree? Not quite.

Neuroimaging evidence has revealed a specialized set of regions within the human brain that respond strongly and selectively to language.

This language system seems to be distinct from regions that are linked to our ability to plan, remember, reminisce on past and future, reason in social situations, experience empathy, make moral decisions, and construct one’s self-image. Thus, vast portions of our everyday cognitive experiences appear to be unrelated to language per se.

But what about Sue? Can she really think the way we do?

While we cannot directly measure what it’s like to think like a neurotypical adult, we can probe Sue’s cognitive abilities by asking her to perform a variety of different tasks. Turns out, patients with global aphasia can solve arithmetic problems, reason about intentions of others, and engage in complex causal reasoning tasks. They can tell whether a drawing depicts a real-life event and laugh when in doesn’t. Some of them play chess in their spare time. Some even engage in creative tasks – a composer Vissarion Shebalin continued to write music even after a stroke that left him severely aphasic.

Some readers might find these results surprising, given that their own thoughts seem to be tied to language so closely. If you find yourself in that category, I have a surprise for you – research has established that not everybody has inner speech experiences. A bilingual friend of mine sometimes gets asked if she thinks in English or Polish, but she doesn’t quite get the question (“how can you think in a language?”). Another friend of mine claims that he “thinks in landscapes,” a sentiment that conveys the pictorial nature of some people’s thoughts. Therefore, even inner speech does not appear to be necessary for thought.

Have we solved the mystery then? Can we claim that language and thought are completely independent and Bertrand Russell was wrong? Only to some extent. We have shown that damage to the language system within an adult human brain leaves most other cognitive functions intact. However, when it comes to the language-thought link across the entire lifespan, the picture is far less clear. While available evidence is scarce, it does indicate that some of the cognitive functions discussed above are, at least to some extent, acquired through language.

Perhaps the clearest case is numbers. There are certain tribes around the world whose languages do not have number words – some might only have words for one through five (Munduruku), and some won’t even have those (Pirahã). Speakers of Pirahã have been shown to make mistakes on one-to-one matching tasks (“get as many sticks as there are balls”), suggesting that language plays an important role in bootstrapping exact number manipulations.

Another way to examine the influence of language on cognition over time is by studying cases when language access is delayed. Deaf children born into hearing families often do not get exposure to sign languages for the first few months or even years of life; such language deprivation has been shown to impair their ability to engage in social interactions and reason about the intentions of others. Thus, while the language system may not be directly involved in the process of thinking, it is crucial for acquiring enough information to properly set up various cognitive domains.

Even after her stroke, our patient Sue will have access to a wide range of cognitive abilities. She will be able to think by drawing on neural systems underlying many non-linguistic skills, such as numerical cognition, planning, and social reasoning. It is worth bearing in mind, however, that at least some of those systems might have relied on language back when Sue was a child. While the static view of the human mind suggests that language and thought are largely disconnected, the dynamic view hints at a rich nature of language-thought interactions across development.

_____

Do you have a question for The Brain? Ask it here.

Alumnus gives MIT $4.5 million to study effects of cannabis on the brain

by MIT School of Science |

The following news is adapted from a press release issued in conjunction with Harvard Medical School.

Charles R. Broderick, an alumnus of MIT and Harvard University, has made gifts to both alma maters to support fundamental research into the effects of cannabis on the brain and behavior.

The gifts, totaling $9 million, represent the largest donation to date to support independent research on the science of cannabinoids. The donation will allow experts in the fields of neuroscience and biomedicine at MIT and Harvard Medical School to conduct research that may ultimately help unravel the biology of cannabinoids, illuminate their effects on the human brain, catalyze treatments, and inform evidence-based clinical guidelines, societal policies, and regulation of cannabis.

Lagging behind legislation

With the increasing use of cannabis both for medicinal and recreational purposes, there is a growing concern about critical gaps in knowledge.

In 2017, the National Academies of Sciences, Engineering, and Medicine issued a report calling upon philanthropic organizations, private companies, public agencies and others to develop a “comprehensive evidence base” on the short- and long-term health effects — both beneficial and harmful — of cannabis use.

“Our desire is to fill the research void that currently exists in the science of cannabis,” says Broderick, who was an early investor in Canada’s medical marijuana market.

Broderick is the founder of Uji Capital LLC, a family office focused on quantitative opportunities in global equity capital markets. Identifying the growth of the Canadian legal cannabis market as a strategic investment opportunity, Broderick took equity positions in Tweed Marijuana Inc. and Aphria Inc., which have since grown into two of North America’s most successful cannabis companies. Subsequently, Broderick made a private investment in and served as a board member for Tokyo Smoke, a cannabis brand portfolio, which merged in 2017 to create Hiku Brands, where he served as chairman. Hiku Brands was acquired by Canopy Growth Corp. in 2018.

Through the Broderick gifts to Harvard Medical School and MIT’s School of Science through the Picower Institute for Learning and Memory and the McGovern Institute for Brain Research, the Broderick funds will support independent studies of the neurobiology of cannabis; its effects on brain development, various organ systems and overall health, including treatment and therapeutic contexts; and cognitive, behavioral and social ramifications.

“I want to destigmatize the conversation around cannabis — and, in part, that means providing facts to the medical community, as well as the general public,” says Broderick, who argues that independent research needs to form the basis for policy discussions, regardless of whether it is good for business. “Then we’re all working from the same information. We need to replace rhetoric with research.”

MIT: Focused on brain health and function

The gift to MIT from Broderick will provide $4.5 million over three years to support independent research for four scientists at the McGovern and Picower institutes.

Two of these researchers — John Gabrieli, the Grover Hermann Professor of Health Sciences and Technology, a professor of brain and cognitive sciences, and a member of MIT’s McGovern Institute for Brain Research; and Myriam Heiman, the Latham Family Associate Professor of Neuroscience at the Picower Institute — will separately explore the relationship between cannabis and schizophrenia.

Gabrieli, who directs the Martinos Imaging Center at MIT, will monitor any potential therapeutic value of cannabis for adults with schizophrenia using fMRI scans and behavioral studies.

“The ultimate goal is to improve brain health and wellbeing,” says Gabrieli. “And we have to make informed decisions on the way to this goal, wherever the science leads us. We need more data.”

Heiman, who is a molecular neuroscientist, will study how chronic exposure to phytocannabinoid molecules THC and CBD may alter the developmental molecular trajectories of cell types implicated in schizophrenia.

“Our lab’s research may provide insight into why several emerging lines of evidence suggest that adolescent cannabis use can be associated with adverse outcomes not seen in adults,” says Heiman.

In addition to these studies, Gabrieli also hopes to investigate whether cannabis can have therapeutic value for autism spectrum disorders, and Heiman plans to look at whether cannabis can have therapeutic value for Huntington’s disease.

MIT Institute Professor Ann Graybiel has proposed to study the cannabinoid 1 (CB1) receptor, which mediates many of the effects of cannabinoids. Her team recently found that CB1 receptors are tightly linked to dopamine — a neurotransmitter that affects both mood and motivation. Graybiel, who is also a member of the McGovern Institute, will examine how CB1 receptors in the striatum, a deep brain structure implicated in learning and habit formation, may influence dopamine release in the brain. These findings will be important for understanding the effects of cannabis on casual users, as well as its relationship to addictive states and neuropsychiatric disorders.

Earl Miller, Picower Professor of Neuroscience at the Picower Institute, will study effects of cannabinoids on both attention and working memory. His lab has recently formulated a model of working memory and unlocked how anesthetics reduce consciousness, showing in both cases a key role in the brain’s frontal cortex for brain rhythms, or the synchronous firing of neurons. He will observe how these rhythms may be affected by cannabis use — findings that may be able to shed light on tasks like driving where maintenance of attention is especially crucial.

Harvard Medical School: Mobilizing basic scientists and clinicians to solve an acute biomedical challenge 

The Broderick gift provides $4.5 million to establish the Charles R. Broderick Phytocannabinoid Research Initiative at Harvard Medical School, funding basic, translational and clinical research across the HMS community to generate fundamental insights about the effects of cannabinoids on brain function, various organ systems, and overall health.

The research initiative will span basic science and clinical disciplines, ranging from neurobiology and immunology to psychiatry and neurology, taking advantage of the combined expertise of some 30 basic scientists and clinicians across the school and its affiliated hospitals.

The epicenter of these research efforts will be the Department of Neurobiology under the leadership of Bruce Bean and Wade Regehr.

“I am excited by Bob’s commitment to cannabinoid science,” says Regehr, professor of neurobiology in the Blavatnik Institute at Harvard Medical School. “The research efforts enabled by Bob’s vision set the stage for unraveling some of the most confounding mysteries of cannabinoids and their effects on the brain and various organ systems.”

Bean, Regehr, and fellow neurobiologists Rachel Wilson and Bernardo Sabatini, for example, focus on understanding the basic biology of the cannabinoid system, which includes hundreds of plant and synthetic compounds as well as naturally occurring cannabinoids made in the brain.

Cannabinoid compounds activate a variety of brain receptors, and the downstream biological effects of this activation are astoundingly complex, varying by age and sex, and complicated by a person’s physiologic condition and overall health. This complexity and high degree of variability in individual biology has hampered scientific understanding of the positive and negative effects of cannabis on the human body. Bean, Regehr, and colleagues have already made critical insights showing how cannabinoids influence cell-to-cell communication in the brain.

“Even though cannabis products are now widely available, and some used clinically, we still understand remarkably little about how they influence brain function and neuronal circuits in the brain,” says Bean, the Robert Winthrop Professor of Neurobiology in the Blavatnik Institute at HMS. “This gift will allow us to conduct critical research into the neurobiology of cannabinoids, which may ultimately inform new approaches for the treatment of pain, epilepsy, sleep and mood disorders, and more.”

To propel research findings from lab to clinic, basic scientists from HMS will partner with clinicians from Harvard-affiliated hospitals, bringing together clinicians and scientists from disciplines including cardiology, vascular medicine, neurology, and immunology in an effort to glean a deeper and more nuanced understanding of cannabinoids’ effects on various organ systems and the body as a whole, rather than just on isolated organs.

For example, Bean and colleague Gary Yellen, who are studying the mechanisms of action of antiepileptic drugs, have become interested in the effects of cannabinoids on epilepsy, an interest they share with Elizabeth Thiele, director of the pediatric epilepsy program at Massachusetts General Hospital. Thiele is a pioneer in the use of cannabidiol for the treatment of drug-resistant forms of epilepsy. Despite proven clinical efficacy and recent FDA approval for rare childhood epilepsies, researchers still do not know exactly how cannabidiol quiets the misfiring brain cells of patients with the seizure disorder. Understanding its mechanism of action could help in developing new agents for treating other forms of epilepsy and other neurologic disorders.

Recurrent architecture enhances object recognition in brain and AI

by Sabbi Lall |

Your ability to recognize objects is remarkable. If you see a cup under unusual lighting or from unexpected directions, there’s a good chance that your brain will still compute that it is a cup. Such precise object recognition is one holy grail for AI developers, such as those improving self-driving car navigation. While modeling primate object recognition in the visual cortex has revolutionized artificial visual recognition systems, current deep learning systems are simplified, and fail to recognize some objects that are child’s play for primates such as humans. In findings published in Nature Neuroscience, McGovern Investigator James DiCarlo and colleagues have found evidence that feedback improves recognition of hard-to-recognize objects in the primate brain, and that adding feedback circuitry also improves the performance of artificial neural network systems used for vision applications.

Deep convolutional neural networks (DCNN) are currently the most successful models for accurately recognizing objects on a fast timescale (<100 ms) and have a general architecture inspired by the primate ventral visual stream, cortical regions that progressively build an accessible and refined representation of viewed objects. Most DCNNs are simple in comparison to the primate ventral stream however.

“For a long period of time, we were far from an model-based understanding. Thus our field got started on this quest by modeling visual recognition as a feedforward process,” explains senior author DiCarlo, who is also the head of MIT’s Department of Brain and Cognitive Sciences and Research Co-Leader in the Center for Brains, Minds, and Machines (CBMM). “However, we know there are recurrent anatomical connections in brain regions linked to object recognition.”

Think of feedforward DCNNs and the portion of the visual system that first attempts to capture objects as a subway line that runs forward through a series of stations. The extra, recurrent brain networks are instead like the streets above, interconnected and not unidirectional. Because it only takes about 200 ms for the brain to recognize an object quite accurately, it was unclear if these recurrent interconnections in the brain had any role at all in core object recognition. For example, perhaps those recurrent connections are only in place to keep the visual system in tune over long periods of time. For example, the return gutters of the streets help slowly clear it of water and trash, but are not strictly needed to quickly move people from one end of town to the other. DiCarlo, along with lead author and CBMM postdoc Kohitij Kar, set out to test whether a subtle role of recurrent operations in rapid visual object recognition was being overlooked.

Challenging recognition

The authors first needed to identify objects that are trivially decoded by the primate brain, but are challenging for artificial systems. Rather than trying to guess why deep learning was having problems recognizing an object (is it due to clutter in the image? a misleading shadow?), the authors took an unbiased approach that turned out to be critical.

Kar explained further that “we realized that AI-models actually don’t have problems with every image where an object is occluded or in clutter. Humans trying to guess why AI models were challenged turned out to be holding us back.”

Instead, the authors presented the deep learning system, as well as monkeys and humans, with images, homing in on “challenge images” where the primates could easily recognize the objects in those images, but a feed forward DCNN ran into problems. When they, and others, added appropriate recurrent processing to these DCNNs, object recognition in challenge images suddenly became a breeze.

Processing times

Kar used neural recording methods with very high spatial and temporal precision to whether these images were really so trivial for primates. Remarkably, they found that though challenge images had initially appeared to be child’s play to the human brain, they actually involve extra neural processing time (about additional 30 milliseconds), suggesting that recurrent loops operate in our brain too.

 “What the computer vision community has recently achieved by stacking more and more layers onto artificial neural networks, evolution has achieved through a brain architecture with recurrent connections.” — Kohitij Kar

Diane Beck, Professor of Psychology and Co-chair of the Intelligent Systems Theme at the Beckman Institute and not an author on the study, explained further. “Since entirely feed forward deep convolutional nets are now remarkably good at predicting primate brain activity, it raised questions about the role of feedback connections in the primate brain. This study shows that, yes, feedback connections are very likely playing a role in object recognition after all.”

What does this mean for a self-driving car? It shows that deep learning architectures involved in object recognition need recurrent components if they are to match the primate brain, and also indicates how to operationalize this procedure for the next generation of intelligent machines.

“Recurrent models offer predictions of neural activity and behavior over time,” says Kar. “We may now be able to model more involved tasks. Perhaps one day, the systems will not only recognize an object, such as a person, but also perform cognitive tasks that the human brain so easily manages, such as understanding the emotions of other people.”

This work was supported by the Office of Naval Research grant MURI-114407 (J.J.D.). Center for Brains, Minds, and Machines (CBMM) funded by NSF STC award CCF-1231216 (K.K.).

Why is the brain shaped like it is?

by Sabbi Lall |

The human brain has a very striking shape, and one feature stands out large and clear: the cerebral cortex with its stereotyped pattern of gyri (folds and convolutions) and sulci (fissures and depressions). This characteristic folded shape of the cortex is a major innovation in evolution that allowed an increase in the size and complexity of the human brain.

How the brain adopts these complex folds is surprisingly unclear, but probably involves both shape changes and movement of cells. Mechanical constraints within the overall tissue, and imposed by surrounding tissues also contribute to the ultimate shape: the brain has to fit into the skull after all. McGovern postdoc Jonathan Wilde has a long-term interest in studying how the brain develops, and explained to us how the shape of the brain initially arises.

In the case of humans, our historical reliance upon intelligence has driven a massive expansion of the cerebral cortex.

“Believe it or not, all vertebrate brains begin as a flat sheet of epithelial cells that folds upon itself to form a tube,” explains Wilde. “This neural tube is made up of a single layer of neural stem cells that go through a rapid and highly orchestrated process of expansion and differentiation, giving rise to all of the neurons in the brain. Throughout the first steps of development, the brains of most vertebrates are indistinguishable from one another, but the final shape of the brain is highly dependent upon the organism and primarily reflects that organism’s lifestyle, environment, and cognitive demands.”

So essentially, the brain starts off as a similar shape for creatures with spinal cords. But why is the human brain such a distinct shape?

“In the case of humans,” explains Wilde, “our historical reliance upon intelligence has driven a massive expansion of the cerebral cortex, which is the primary brain structure responsible for critical thinking and higher cognitive abilities. Accordingly, the human cortex is strikingly large and covered in a labyrinth of folds that serve to increase its surface area and computational power.”

The anatomical shape of the human brain is striking, but it also helps researchers to map a hidden functional atlas: specific brain regions that selectively activate in fMRI when you see a face, scene, hear music and a variety of other tasks. I asked former McGovern graduate student, and current postdoc at Boston Children’s Hospital, Hilary Richardson, for her perspective on this more hidden structure in the brain and how it relates to brain shape.

Illustration of person rappelling into the brain's sylvian fissure.
The Sylvian fissure is a prominent groove on each side of the brain that separates the frontal and parietal lobes from the temporaal lobe. McGovern researchers are studying a region near the right Sylvian fissure, called the rTPJ, which is involved in thinking about what another person is thinking. Image: Joe Laney

“One of the most fascinating aspects of brain shape is how similar it is across individuals, even very young infants and children,” explains Richardson. “Despite the dramatic cognitive changes that happen across childhood, the shape of the brain is remarkably consistent. Given this, one open question is what kinds of neural changes support cognitive development. For example, while the anatomical shape and size of the rTPJ seems to stay the same across childhood, its response becomes more specialized to information about mental states – beliefs, desires, and emotions – as children get older. One intriguing hypothesis is that this specialization helps support social development in childhood.”

We’ll end with an ode to a prominent feature of brain shape: the “Sylvian fissure,” a prominent groove on each side of the brain that separates the frontal and parietal lobes from the temporal lobe. Such landmarks in brain shape help orient researchers, and the Sylvian fissure was recently immortalized in this image, from a postcard by illustrator Joe Laney.

______

Do you have a question for The Brain? Ask it here.

 

How our gray matter tackles gray areas

by Gina Vitale | MIT News correspondent |

When Katie O’Nell’s high school biology teacher showed a NOVA video on epigenetics after the AP exam, he was mostly trying to fill time. But for O’Nell, the video sparked a whole new area of curiosity.

She was fascinated by the idea that certain genes could be turned on and off, controlling what traits or processes were expressed without actually editing the genetic code itself. She was further excited about what this process could mean for the human mind.

But upon starting at MIT, she realized that she was less interested in the cellular level of neuroscience and more fascinated by bigger questions, such as, what makes certain people generous toward certain others? What’s the neuroscience behind morality?

“College is a time you can learn about anything you want, and what I want to know is why humans are really, really wacky,” she says. “We’re dumb, we make super irrational decisions, it makes no sense. Sometimes it’s beautiful, sometimes it’s awful.”

O’Nell, a senior majoring in brain and cognitive sciences, is one of five MIT students to have received a Marshall Scholarship this year. Her quest to understand the intricacies of the wacky human brain will not be limited to any one continent. She will be using the funding to earn her master’s in experimental psychology at Oxford University.

Chocolate milk and the mouse brain

O’Nell’s first neuroscience-related research experience at MIT took place during her sophomore and junior year, in the lab of Institute Professor Ann Graybiel at the McGovern Institute.

The research studied the neurological components of risk-vs-reward decision making, using a key ingredient: chocolate milk. In the experiments, mice were given two options — they could go toward the richer, sweeter chocolate milk, but they would also have to endure a brighter light. Or, they could go toward a more watered-down chocolate milk, with the benefit of a softer light. All the while, a fluorescence microscope tracked when certain cell types were being activated.

“I think that’s probably the closest thing I’ve ever had to a spiritual experience … watching this mouse in this maze deciding what to do, and watching the cells light up on the screen. You can see single-cell evidence of cognition going on. That’s just the coolest thing.”

In her junior spring, O’Nell delved even deeper into questions of morality in the lab of Professor Rebecca Saxe. Her research there centers on how the human brain parses people’s identities and emotional states from their faces alone, and how those computations are related to each other. Part of what interests O’Nell is the fact that we are constantly making decisions, about ourselves and others, with limited information.

“We’re always solving under uncertainty,” she says. “And our brain does it so well, in so many ways.”

International intrigue

Outside of class, O’Nell has no shortage of things to do. For starters, she has been serving as an associate advisor for a first-year seminar since the fall of her sophomore year.

“Basically it’s my job to sit in on a seminar and bully them into not taking seven classes at a time, and reminding them that yes, your first 8.01 exam is tomorrow,” she says with a laugh.

She has also continued an activity she was passionate about in high school — Model United Nations. One of the most fun parts for her is serving on the Historical Crisis Committee, in which delegates must try to figure out a way to solve a real historical problem, like the Cuban Missile Crisis or the French and Indian War.

“This year they failed and the world was a nuclear wasteland,” she says. “Last year, I don’t entirely know how this happened, but France decided that they wanted to abandon the North American theater entirely and just took over all of Britain’s holdings in India.”

She’s also part of an MIT program called the Addir Interfaith Fellowship, in which a small group of people meet each week and discuss a topic related to religion and spirituality. Before joining, she didn’t think it was something she’d be interested in — but after being placed in a first-year class about science and spirituality, she has found discussing religion to be really stimulating. She’s been a part of the group ever since.

O’Nell has also been heavily involved in writing and producing a Mystery Dinner Theater for Campus Preview Weekend, on behalf of her living group J Entry, in MacGregor House. The plot, generally, is MIT-themed — a physics professor might get killed by a swarm of CRISPR nanobots, for instance. When she’s not cooking up murder mysteries, she might be running SAT classes for high school students, playing piano, reading, or spending time with friends. Or, when she needs to go grocery shopping, she’ll be stopping by the Trader Joe’s on Boylston Avenue, as an excuse to visit the Boston Public Library across the street.

Quite excited for the future

O’Nell is excited that the Marshall Scholarship will enable her to live in the country that produced so many of the books she cherished as a kid, like “The Hobbit.” She’s also thrilled to further her research there. However, she jokes that she still needs to get some of the lingo down.

“I need to learn how to use the word ‘quite’ correctly. Because I overuse it in the American way,” she says.

Her master’s research will largely expand on the principles she’s been examining in the Saxe lab. Questions of morality, processing, and social interaction are where she aims to focus her attention.

“My master’s project is going to be basically taking a look at whether how difficult it is for you to determine someone else’s facial expression changes how generous you are with people,” she explains.

After that, she hopes to follow the standard research track of earning a PhD, doing postdoctoral research, and then entering academia as a professor and researcher. Teaching and researching, she says, are two of her favorite things — she’s excited to have the chance to do both at the same time. But that’s a few years ahead. Right now, she hopes to use her time in England to learn all she can about the deeper functions of the brain, with or without chocolate milk.

3Q: The interface between art and neuroscience

by Connie Blaszczyk | Center for Art, Science, and Technology |

CBMM postdoc Sarah Schwettman

Computational neuroscientist Sarah Schwettmann, who works in the Center for Brains, Minds, and Machines at the McGovern Institute, is one of three instructors behind the cross-disciplinary course 9.S52/9.S916 (Vision in Art and Neuroscience), which introduces students to core concepts in visual perception through the lenses of art and neuroscience.

Supported by a faculty grant from the Center for Art, Science and Technology at MIT (CAST) for the past two years, the class is led by Pawan Sinha, a professor of vision and computational neuroscience in the Department of Brain and Cognitive Sciences. They are joined in the course by Seth Riskin SM ’89, a light artist and the manager of the MIT Museum Studio and Compton Gallery, where the course is taught. Schwettman discussed the combination of art and science in an educational setting.

Q: How have the three of you approached this cross-disciplinary class in art and neuroscience?

A: Discussions around this intersection often consider what each field has to offer the other. We take a different approach, one I refer to as occupying the gap, or positioning ourselves between the two fields and asking what essential questions underlie them both. One question addresses the nature of the human relationship to the world. The course suggests one answer: This relationship is fundamentally creative, from the brain’s interpretation of incoming sensory data in perception, to the explicit construction of experiential worlds in art.

Neuroscience and art, therefore, each provide a set of tools for investigating different levels of the constructive process. Through neuroscience, we develop a specific understanding of the models of the world that the brain uses to make sense of incoming visual data. With articulation of those models, we can engineer types of inputs that interact with visual processing architecture in particularly exquisite ways, and do so reliably, giving artists a toolkit for remixing and modulating experience. In the studio component of the course, we experiment with this toolkit and collectively move it forward.

While designing the course, Pawan, Seth, and I found that we were each addressing a similar set of questions, the same that motivate the class, through our own research and practice. In parallel to computational vision research, Professor Sinha leads a humanitarian initiative called Project Prakash, which provides treatment to blind children in India and explores the development of vision following the restoration of sight. Where does structure in perception originate? As an artist in the MIT Museum Studio, Seth works with articulated light to sculpt structured visual worlds out of darkness. I also live on this interface where the brain meets the world — my research in the Department of Brain and Cognitive Sciences examines the neural basis of mental models for simulating physics. Linking our work in the course is an experiment in synthesis.

Q: What current research in vision, neuroscience, and art are being explored at MIT, and how does the class connect it to hands-on practice?

A: Our brains build a rich world of experience and expectation from limited and noisy sensory data with infinite potential interpretations. In perception research, we seek to discover how the brain finds more meaning in incoming data than is explained by the signal alone. Work being done at MIT around generative models addresses this, for instance in the labs of Josh Tenenbaum and Josh McDermott in the Department of Brain and Cognitive Sciences. Researchers present an ambiguous visual or auditory stimulus and by probing someone’s perceptual interpretation, they get a handle on the structures that the mind generates to interpret incoming data, and they can begin to build computational models of the process.

In Vision in Art and Neuroscience, we focus on the experiential as well as the experimental, probing the perceiver’s experience of structure-generating process—perceiving perception itself.

As instructors, we face the pedagogical question: what exercises, in the studio, can evoke so striking an experience of students’ own perception that cutting edge research takes on new meaning, understood in the immediacy of seeing? Later in the semester, students face a similar question as artists: How can one create visual environments where viewers experience their own perceptual processing at work? Done well, this experience becomes the artwork itself. Early in the course, students explore the Ganzfeld effect, popularized by artist James Turrell, where the viewer is exposed to an unstructured visual field of uniform illumination. In this experience, one feels the mind struggling to fit models of the world to unstructured input, and attempting this over and over again — an interpretation process which often goes unnoticed when input structure is expected by visual processing architecture. The progression of the course modules follows the hierarchy of visual processing in the brain, which builds increasingly complex interpretations of visual inputs, from brightness and edges to depth, color, and recognizable form.

MIT students first encounter those concepts in the seminar component of the course at the beginning of each week. Later in the week, students translate findings into experimental approaches in the studio. We work with light directly, from introducing a single pinpoint of light into an otherwise completely dark room, to building intricate environments using programmable electronics. Students begin to take this work into their own hands, in small groups and individually, culminating in final projects for exhibition. These exhibitions are truly a highlight of the course. They’re often one of the first times that students have built and shown artworks. That’s been a gift to share with the broader MIT community, and a great learning experience for students and instructors alike.

Q: How has that approach been received by the MIT community?

A: What we’re doing has resonated across disciplines: In addition to neuroscience, we have students and researchers joining us from computer science, mechanical engineering, mathematics, the Media Lab, and ACT (the Program in Art, Culture, and Technology). The course is growing into something larger, a community of practice interested in applying the scientific methodology we develop to study the world, to probe experience, and to articulate models for its generation and replication.

With a mix of undergraduates, graduates, faculty, and artists, we’ve put together installations and symposia — including three on campus so far. The first of these, “Perceiving Perception,” also led to a weekly open studio night where students and collaborators convene for project work. Our second exhibition, “Dessert of the Real,” is on display this spring in the Compton Gallery. This April we’re organizing a symposium in the studio featuring neuroscientists, computer scientists, artists and researchers from MIT and Harvard. We’re reaching beyond campus as well, through off-site installations, collaborations with museums — including the Metropolitan Museum of Art and the Peabody Essex Museum — and a partnership with the ZERO Group in Germany.

We’re eager to involve a broad network of collaborators. It’s an exciting moment in the fields of neuroscience and computing; there is great energy to build technologies that perceive the world like humans do. We stress on the first day of class that perception is a fundamentally creative act. We see the potential for models of perception to themselves be tools for scaling and translating creativity across domains, and for building a deeply creative relationship to our environment.

Halassa named Max Planck Fellow

by Sabbi Lall |

Michael Halassa was just appointed as one of the newest Max Planck Fellows. His appointment comes through the Max Planck Florida Institute for Neuroscience (MPFI), which aims to forge collaborations between exceptional neuroscientists from around the world to answer fundamental questions about brain development and function. The Max Planck Society selects cutting edge, active researchers from other institutions to fellow positions for a five-year period to promote interactions and synergies. While the program is a longstanding feature of the Max Planck Society, Halassa, and fellow appointee Yi Guo of the University of California, Santa Cruz, are the first selected fellows that are based at U.S. institutions.

Michael Halassa is an associate investigator at the McGovern Institute and an assistant professor in the Department of Brain and Cognitive Sciences at MIT. Halassa’s research focuses on the neural architectures that underlie complex cognitive processes. He is particularly interested in goal-directed attention, our ability to rapidly switch attentional focus based on high level objectives. For example, when you are in a roomful of colleagues, the mention of your name in a distant conversation can quickly trigger your ‘mind’s ear’ to eavesdrop into that conversation. This contrasts with hearing a name that sounds like yours on television, which does not usually grab your attention in the same way. In certain mental disorders such as schizophrenia, the ability to generate such high-level objectives, while also accounting for context, is perturbed. Recent evidence strongly suggests that impaired function of the prefrontal cortex and its interactions with a region of the brain called the thalamus may be altered in such disorders. It is this thalamocortical network that Halassa has been studying in mice, where his group has uncovered how the thalamus supports the ability of the prefrontal cortex to generate context-appropriate attentional signals.

The fellowship will support extending Halassa’s work into the tree shrew (Tupaia belangeri), which has been shown to have advanced cognitive abilities compared to mice while also offering many of the circuit-interrogation tools that make the mouse an attractive experimental model.

The Max Planck Florida Institute for Neuroscience (MPFI), a not-for-profit research organization, is part of the world-renowned Max Planck Society, Germany’s most successful research organization. The Max Planck Society was founded in 1911, and comprises 84 institutes and research facilities. While primarily located in Germany, there are 4 institutes and one research facility located aboard, including the Florida Institute that Halassa will collaborate with. The fellow positions were created with the goal of increasing interactions between the Max Planck Society and its institutes with faculty engaged in active research at other universities and institutions, which with this appointment now include MIT.

How does the brain focus?

by Sabbi Lall |

This is a very interesting question, and one that researchers at the McGovern Institute for Brain Research are actively pursuing. It’s also important for understanding what happens in conditions such as ADHD. There are constant distractions in the world, a cacophony of noise and visual stimulation. How and where we focus our attention, and what the brain attends to vs. treating as background information, is a big question in neuroscience. Thanks to work from researchers, including Robert Desimone, we understand quite a bit about how this works in the visual system in particular. What his lab has found is that when we pay attention to something specific, neurons in the visual cortex responding to the object we’re focusing upon fire in synchrony, whereas those responding to irrelevant information become suppressed. It’s almost as if this synchrony “increases the volume” so that the responding neurons rise above general noise.

Synchronized activity of neurons occurs as they oscillate together at a particular frequency, but the frequency of oscillation really matters when it comes to attention and focus vs. inattention and distraction. To find out more about this, I asked a postdoc in the Desimone lab, Yasaman Bagherzadeh about the role of different “brainwaves,” or oscillations at different frequencies, in attention.

“Studies in humans have shown that enhanced synchrony between neurons in the alpha range –8–12 Hz— is actually associated with inattention and distracting information,” explains Bagherzadeh, “whereas enhanced gamma synchrony (about 30-150 Hz) is associated with attention and focus on a target. For example, when a stimulus (through the ears or eyes) or its location (left vs. right) is intentionally ignored, this is preceded by a relative increase in alpha power, while a stimulus you’re attending to is linked to an increase in gamma power.”

Attention in the Desimone lab (no pun intended) has also recently been focused on covert attention. This type of spatial attention was traditionally thought to occur through a mental shift without a glance, but the Desimone lab recently found that even during these mental shifts, animal sneakily glance at objects that attention becomes focused on. Think now of something you know is nearby (a cup of coffee for example), but not in the center of your field of vision. Chances are that you just sneakily glanced at that object.

Previously these sneaky glances/small eye movements, called microsaccades (MS for short), were considered to be involuntary movements without any functional role. However, in the recent Desimone lab study, it was found that a MS significantly modulates neural activity during the attention period. This means that when you glance at something, even sneakily, it is intimately linked to attention. In other words, when it comes to spatial attention, eye movements seem to play a significant role.

Various questions arise about the mechanisms of spatial attention as a result this study, as outlined by Karthik Srinivasan, a postdoctoral associate in the Desimone lab.

“How are eye movement signals and attentional processing coordinated? What’s the role of the different frequencies of oscillation for such coordination? Is there a role for them or are they just the frequency domain representation (i.e., an epiphenomenon) of a temporal/dynamical process? Is attention a sustained process or rhythmic or something more dynamic?” Srinivasan lists some of the questions that come out of his study and goes on to explain the implications of the study further. “It is hard to believe that covert attention is a sustained process (the so-called ‘spotlight theory of attention’), given that neural activity during the attention period can be modulated by covert glances. A few recent studies have supported the idea that attention is a rhythmic process that can be uncoupled from eye movements. While this is an idea made attractive by its simplicity, it’s clear that small glances can affect neural activity related to attention, and MS are not rhythmic. More work is thus needed to get to a more unified theory that accounts for all of the data out there related to eye movements and their close link to attention.”

Answering some of the questions that Bagherzadeh, Srinivasan, and others are pursuing in the Desimone lab, both experimentally and theoretically, will clear up some of the issues above, and improve our understanding of how the brain focuses attention.

Do you have a question for The Brain? Ask it here.

 

How motion conveys emotion in the face

by Sabbi Lall |

While a static emoji can stand in for emotion, in real life we are constantly reading into the feelings of others through subtle facial movements. The lift of an eyebrow, the flicker around the lips as a smile emerges, a subtle change around the eyes (or the sudden rolling of the eyes), are all changes that feed into our ability to understand the emotional state, and the attitude, of others towards us. Ben Deen and Rebecca Saxe have now monitored changes in brain activity as subjects followed face movements in movies of avatars. Their findings argue that we can generalize across individual face part movements in other people, but that a particular cortical region, the face-responsive superior temporal sulcus (fSTS), is also responding to isolated movements of individual face parts. Indeed, the fSTS seems to be tied to kinematics, individual face part movement, more than the implied emotional cause of that movement.

We know that the brain responds to dynamic changes in facial expression, and that these are associated with activity in the fSTS, but how do calculations of these movements play out in the brain?

Do we understand emotional changes by adding up individual features (lifting of eyebrows + rounding of mouth= surprise), or are we assessing the entire face in a more holistic way that results in more generalized representations? McGovern Investigator Rebecca Saxe and her graduate student Ben Deen set out to answer this question using behavioral analysis and brain imaging, specifically fMRI.

“We had a good sense of what stimuli the fSTS responds strongly to,” explains Ben Deen, “but didn’t really have any sense of how those inputs are processed in the region – what sorts of features are represented, whether the representation is more abstract or more tied to visual features, etc. The hope was to use multivoxel pattern analysis, which has proven to be a remarkably useful method for characterizing representational content, to address these questions and get a better sense of what the region is actually doing.”

Facial movements were conveyed to subjects using animated “avatars.” By presenting avatars that made isolated eye and eyebrow movements (brow raise, eye closing, eye roll, scowl) or mouth movements (smile, frown, mouth opening, snarl), as well as composites of these movements, the researchers were able to assess whether our interpretation of the latter is distinct from the sum of its parts. To do this, Deen and Saxe first took a behavioral approach where people reported on what combinations of eye and mouth movements in a whole avatar face, or one where the top and bottom parts of the face were misaligned. What they found was that movement in the mouth region can influence perception of movement in the eye region, arguably due to some level of holistic processing. The authors then asked whether there were cortical differences upon viewing isolated vs. combined face part movements. They found that changes in fSTS, but not other brain regions, had patterns of activity that seemed to discriminate between different facial movements. Indeed, they could decode which part of the avatar’s face is being perceived as moving from fSTS activity. The researchers could even model the fSTS response to combined features linearly based on the response to individual face parts. In short, though the behavorial data indicate that there is holistic processing of complex facial movement, it is also clear that isolated parts-based representations are also present, a sort of intermediate state.

As part of this work, Deen and Saxe took the important step of pre-registering their experimental parameters, before collecting any data, at the Open Science Framework. This step allows others to more easily reproduce the analysis they conducted, since all parameters (the task that subjects are carrying out, the number of subjects needed, the rationale for this number, and the scripts used to analyze data) are openly available.

“Preregistration had a big impact on our workflow for the study,” explained Deen. “More of the work was done up front, in coming up with all of the analysis details and agonizing over whether we were choosing the right strategy, before seeing any of the data. When you tie your hands by making these decisions up front, you start thinking much more carefully about them.”

Pre-registration does remove post-hoc researcher subjectivity from the analysis. As an example, because Deen and Saxe predicted that the people would be accurately able to discriminate between faces per se, they decided ahead of the experiment to focus on analyzing reaction time, rather than looking at the collected data and deciding to focus on this number after the fact. This adds to the overall objectivity of the experiment and is increasingly seen as a robust way to conduct such experiments.

Josh McDermott

The Science of Hearing

Hearing enables us to make sense of our whereabouts, understand the emotional state of others, and enjoy musical experiences. Acoustic information relays vital cues about the world—yet much of the sophisticated brain system that decodes this information is poorly understood.

Josh McDermott’s research is in search of foundational principles of sound perception. Groundbreaking discoveries from the McDermott lab have clarified how people hear and process sounds. His research informs new treatments for those with hearing loss, and paves the way for machine systems that emulate the human ability to recognize and interpret sound. McDermott’s lab has also pioneered new approaches for understanding music perception. His lab deconstructs the neural ensembles that allow us to appreciate music, while also studying the often striking variation that can occur across cultures.

Virtual Tour of McDermott Lab