Author: MIT News

Computational model mimics humans’ ability to predict emotions

Anne Trafton |

When interacting with another person, you likely spend part of your time trying to anticipate how they will feel about what you’re saying or doing. This task requires a cognitive skill called theory of mind, which helps us to infer other people’s beliefs, desires, intentions, and emotions.

MIT neuroscientists have now designed a computational model that can predict other people’s emotions — including joy, gratitude, confusion, regret, and embarrassment — approximating human observers’ social intelligence. The model was designed to predict the emotions of people involved in a situation based on the prisoner’s dilemma, a classic game theory scenario in which two people must decide whether to cooperate with their partner or betray them.

To build the model, the researchers incorporated several factors that have been hypothesized to influence people’s emotional reactions, including that person’s desires, their expectations in a particular situation, and whether anyone was watching their actions.

“These are very common, basic intuitions, and what we said is, we can take that very basic grammar and make a model that will learn to predict emotions from those features,” says Rebecca Saxe, the John W. Jarve Professor of Brain and Cognitive Sciences, a member of MIT’s McGovern Institute for Brain Research, and the senior author of the study.

Sean Dae Houlihan PhD ’22, a postdoc at the Neukom Institute for Computational Science at Dartmouth College, is the lead author of the paper, which appears today in Philosophical Transactions A. Other authors include Max Kleiman-Weiner PhD ’18, a postdoc at MIT and Harvard University; Luke Hewitt PhD ’22, a visiting scholar at Stanford University; and Joshua Tenenbaum, a professor of computational cognitive science at MIT and a member of the Center for Brains, Minds, and Machines and MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL).

Predicting emotions

While a great deal of research has gone into training computer models to infer someone’s emotional state based on their facial expression, that is not the most important aspect of human emotional intelligence, Saxe says. Much more important is the ability to predict someone’s emotional response to events before they occur.

“The most important thing about what it is to understand other people’s emotions is to anticipate what other people will feel before the thing has happened,” she says. “If all of our emotional intelligence was reactive, that would be a catastrophe.”

To try to model how human observers make these predictions, the researchers used scenarios taken from a British game show called “Golden Balls.” On the show, contestants are paired up with a pot of $100,000 at stake. After negotiating with their partner, each contestant decides, secretly, whether to split the pool or try to steal it. If both decide to split, they each receive $50,000. If one splits and one steals, the stealer gets the entire pot. If both try to steal, no one gets anything.

Depending on the outcome, contestants may experience a range of emotions — joy and relief if both contestants split, surprise and fury if one’s opponent steals the pot, and perhaps guilt mingled with excitement if one successfully steals.

To create a computational model that can predict these emotions, the researchers designed three separate modules. The first module is trained to infer a person’s preferences and beliefs based on their action, through a process called inverse planning.

“This is an idea that says if you see just a little bit of somebody’s behavior, you can probabilistically infer things about what they wanted and expected in that situation,” Saxe says.

Using this approach, the first module can predict contestants’ motivations based on their actions in the game. For example, if someone decides to split in an attempt to share the pot, it can be inferred that they also expected the other person to split. If someone decides to steal, they may have expected the other person to steal, and didn’t want to be cheated. Or, they may have expected the other person to split and decided to try to take advantage of them.

The model can also integrate knowledge about specific players, such as the contestant’s occupation, to help it infer the players’ most likely motivation.

The second module compares the outcome of the game with what each player wanted and expected to happen. Then, a third module predicts what emotions the contestants may be feeling, based on the outcome and what was known about their expectations. This third module was trained to predict emotions based on predictions from human observers about how contestants would feel after a particular outcome. The authors emphasize that this is a model of human social intelligence, designed to mimic how observers causally reason about each other’s emotions, not a model of how people actually feel.

“From the data, the model learns that what it means, for example, to feel a lot of joy in this situation, is to get what you wanted, to do it by being fair, and to do it without taking advantage,” Saxe says.

Core intuitions

Once the three modules were up and running, the researchers used them on a new dataset from the game show to determine how the models’ emotion predictions compared with the predictions made by human observers. This model performed much better at that task than any previous model of emotion prediction.

The model’s success stems from its incorporation of key factors that the human brain also uses when predicting how someone else will react to a given situation, Saxe says. Those include computations of how a person will evaluate and emotionally react to a situation, based on their desires and expectations, which relate to not only material gain but also how they are viewed by others.

“Our model has those core intuitions, that the mental states underlying emotion are about what you wanted, what you expected, what happened, and who saw. And what people want is not just stuff. They don’t just want money; they want to be fair, but also not to be the sucker, not to be cheated,” she says.

“The researchers have helped build a deeper understanding of how emotions contribute to determining our actions; and then, by flipping their model around, they explain how we can use people’s actions to infer their underlying emotions. This line of work helps us see emotions not just as ‘feelings’ but as playing a crucial, and subtle, role in human social behavior,” says Nick Chater, a professor of behavioral science at the University of Warwick, who was not involved in the study.

In future work, the researchers hope to adapt the model so that it can perform more general predictions based on situations other than the game-show scenario used in this study. They are also working on creating models that can predict what happened in the game based solely on the expression on the faces of the contestants after the results were announced.

The research was funded by the McGovern Institute; the Paul E. and Lilah Newton Brain Science Award; the Center for Brains, Minds, and Machines; the MIT-IBM Watson AI Lab; and the Multidisciplinary University Research Initiative.

Tackling the MIT campus’s top energy consumers, building by building

Anne Trafton |

When staff in MIT’s Department of Facilities would visualize energy use and carbon-associated emissions by campus buildings, Building 46 always stood out — attributed to its energy intensity, which accounted for 8 percent of MIT’s total campus energy use. This high energy draw was not surprising, as the building is home of the Brain and Cognitive Sciences Complex and a large amount of lab space, but it also made the building a perfect candidate for an energy performance audit to seek out potential energy saving opportunities.

This audit revealed that several energy efficiency updates to the building mechanical systems infrastructure, including optimization of the room-by-room ventilation rates, could result in an estimated 35 percent reduction of energy use, which would in turn lower MIT’s total greenhouse gas emissions by an estimated 2 percent — driving toward the Institute’s 2026 goal of net-zero and 2050 goal of elimination of direct campus emissions.

Building energy efficiency projects are not new for MIT. Since 2010, MIT has been engaged in a partnership agreement with utility company Eversource establishing the Efficiency Forward program, empowering MIT to invest in more than 300 energy conservation projects to date and lowering energy consumption on campus for a total calculated savings of approximately 70 million kilowatt hours and 4.2 million therms. But at 418,000 gross square feet, Building 46 is the first energy efficiency project of its size on the campus.

“We’ve never tackled a whole building like this — it’s the first capital project that is technically an energy project,” explains Siobhan Carr, energy efficiency program manager, who was part of the team overseeing the energy audit and lab ventilation performance assessment in the building. “That gives you an idea of the magnitude and complexity of this.”

The project started with the full building energy assessment and lab ventilation risk audit. “We had a team go through every corner of the building and look at every possible opportunity to save energy,” explains Jessica Parks, senior project manager for systems performance and turnover in campus construction. “One of the biggest issues we saw was that there’s a lot of dry lab spaces which are basically offices, but they’re all getting the same ventilation as if they were a high-intensity lab.” Higher ventilation and more frequent air exchange rates draw more energy. By optimizing for the required ventilation rates, there was an opportunity to save energy in nearly every space in the building.

In addition to the optimized ventilation, the project team will convert fume hoods from constant volume to variable volume and install equipment to help the building systems run more efficiently. The team also identified opportunities to work with labs to implement programs such as fume hood hibernation and unoccupied setbacks for temperature and ventilation. As different spaces in the building have varying needs, the energy retrofit will touch all 1,254 spaces in the building — one by one — to implement the different energy measures to reach that estimated 35 percent reduction in energy use.

Although time-consuming and complex, this room-by-room approach has a big benefit in that it has allowed research to continue in the space largely uninterrupted. With a few exceptions, the occupants of Building 46, which include the Department of Brain and Cognitive Sciences, The McGovern Institute for Brain Research, and The Picower Institute for Learning and Memory, have remained in place for the duration of the project. Partners in the MIT Environment, Health and Safety Office are instrumental to this balance of renovations and keeping the building operational during the optimization efforts and are one of several teams across MIT contributing to building efficiency efforts.

The completion date of the building efficiency project is set for 2024, but Carr says that some of the impact of this ongoing work may soon be seen. “We should start to see savings as we move through the building, and we expect to fully realize all of our projected savings a year after completion,” she says, noting that the length of time is required for a year-over-year perspective to see the full reduction in energy use.

The impact of the project goes far beyond the footprint of Building 46 as it offers insights and spurred actions for future projects — including buildings 76 and 68, the number two and three top energy users on campus. Both buildings recently underwent their own energy audits and lab ventilation performance assessments. The energy efficiency team is now crafting a plan for full-building approaches, much like Building 46. “To date, 46 has presented many learning opportunities, such as how to touch every space in a building while research continues, as well as how to overcome challenges encountered when working on existing systems,” explains Parks. “The good news is that we have developed solutions for those challenges and the teams have been proactively implementing those lessons in our other projects.”

Communication has proven to be another key for these large projects where occupants see the work happening and often play a role in answering questions about their unique space. “People are really engaged, they ask questions about the work, and we ask them about the space they’re in every day,” says Parks. “The Building 46 occupants have been wonderful partners as we worked in all of their spaces, which is paving the way for a successful project.”

The release of Fast Forward in 2021 has also made communications easier, notes Carr, who says the plan helps to frame these projects as part of the big picture — not just a construction interruption. “Fast Forward has brought a visibility into what we’re doing within [MIT] Facilities on these buildings,” she says. “It brings more eyes and ears, and people understand that these projects are happening throughout campus and not just in their own space — we’re all working to reduce energy and to reduce greenhouse gas across campus.”

The Energy Efficiency team will continue to apply that big-picture approach as ongoing building efficiency projects on campus are assessed to reach toward a 10 to 15 percent reduction in energy use and corresponding emissions over the next several years.

Scientists discover how mutations in a language gene produce speech deficits

Anne Trafton |

Mutations of a gene called Foxp2 have been linked to a type of speech disorder called apraxia that makes it difficult to produce sequences of sound. A new study from MIT and National Yang Ming Chiao Tung University sheds light on how this gene controls the ability to produce speech.

In a study of mice, the researchers found that mutations in Foxp2 disrupt the formation of dendrites and neuronal synapses in the brain’s striatum, which plays important roles in the control of movement. Mice with these mutations also showed impairments in their ability to produce the high-frequency sounds that they use to communicate with other mice.

Those malfunctions arise because Foxp2 mutations prevent the proper assembly of motor proteins, which move molecules within cells, the researchers found.

“These mice have abnormal vocalizations, and in the striatum there are many cellular abnormalities,” says Ann Graybiel, an MIT Institute Professor, a member of MIT’s McGovern Institute for Brain Research, and an author of the paper. “This was an exciting finding. Who would have thought that a speech problem might come from little motors inside cells?”

Fu-Chin Liu PhD ’91, a professor at National Yang Ming Chiao Tung University in Taiwan, is the senior author of the study, which appears today in the journal Brain. Liu and Graybiel also worked together on a 2016 study of the potential link between Foxp2 and autism spectrum disorder. The lead authors of the new Brain paper are Hsiao-Ying Kuo and Shih-Yun Chen of National Yang Ming Chiao Tung University.

Speech control

Children with Foxp2-associated apraxia tend to begin speaking later than other children, and their speech is often difficult to understand. The disorder is believed to arise from impairments in brain regions, such as the striatum, that control the movements of the lips, mouth, and tongue. Foxp2 is also expressed in the brains of songbirds such as zebra finches and is critical to those birds’ ability to learn songs.

Foxp2 encodes a transcription factor, meaning that it can control the expression of many other target genes. Many species express Foxp2, but humans have a special form of Foxp2. In a 2014 study, Graybiel and colleagues found evidence that the human form of Foxp2, when expressed in mice, allowed the mice to accelerate the switch from declarative to procedural types of learning.

In that study, the researchers showed that mice engineered to express the human version of Foxp2, which differs from the mouse version by only two DNA base pairs, were much better at learning mazes and performing other tasks that require turning repeated actions into behavioral routines. Mice with human-like Foxp2 also had longer dendrites — the slender extensions that help neurons form synapses — in the striatum, which is involved in habit formation as well as motor control.

In the new study, the researchers wanted to explore how the Foxp2 mutation that has been linked with apraxia affects speech production, using ultrasonic vocalizations in mice as a proxy for speech. Many rodents and other animals such as bats produce these vocalizations to communicate with each other.

While previous studies, including the work by Liu and Graybiel in 2016, had suggested that Foxp2 affects dendrite growth and synapse formation, the mechanism for how that occurs was not known. In the new study, led by Liu, the researchers investigated one proposed mechanism, which is that Foxp2 affects motor proteins.

One of these molecular motors is the dynein protein complex, a large cluster of proteins that is responsible for shuttling molecules along microtubule scaffolds within cells.

“All kinds of molecules get shunted around to different places in our cells, and that’s certainly true of neurons,” Graybiel says. “There’s an army of tiny molecules that move molecules around in the cytoplasm or put them into the membrane. In a neuron, they may send molecules from the cell body all the way down the axons.”

A delicate balance

The dynein complex is made up of several other proteins. The most important of these is a protein called dynactin1, which interacts with microtubules, enabling the dynein motor to move along microtubules. In the new study, the researchers found that dynactin1 is one of the major targets of the Foxp2 transcription factor.

The researchers focused on the striatum, one of the regions where Foxp2 is most often found, and showed that the mutated version of Foxp2 is unable to suppress dynactin1 production. Without that brake in place, cells generate too much dynactin1. This upsets the delicate balance of dynein-dynactin1, which prevents the dynein motor from moving along microtubules.

Those motors are needed to shuttle molecules that are necessary for dendrite growth and synapse formation on dendrites. With those molecules stranded in the cell body, neurons are unable to form synapses to generate the proper electrophysiological signals they need to make speech production possible.

Mice with the mutated version of Foxp2 had abnormal ultrasonic vocalizations, which typically have a frequency of around 22 to 50 kilohertz. The researchers showed that they could reverse these vocalization impairments and the deficits in the molecular motor activity, dendritic growth, and electrophysiological activity by turning down the gene that encodes dynactin1.

Mutations of Foxp2 can also contribute to autism spectrum disorders and Huntington’s disease, through mechanisms that Liu and Graybiel previously studied in their 2016 paper and that many other research groups are now exploring. Liu’s lab is also investigating the potential role of abnormal Foxp2 expression in the subthalamic nucleus of the brain as a possible factor in Parkinson’s disease.

The research was funded by the Ministry of Science and Technology of Taiwan, the Ministry of Education of Taiwan, the U.S. National Institute of Mental Health, the Saks Kavanaugh Foundation, the Kristin R. Pressman and Jessica J. Pourian ’13 Fund, and Stephen and Anne Kott.

2023 MacVicar Faculty Fellows named

Anne Trafton |

The Office of the Vice Chancellor and the Registrar’s Office have announced this year’s Margaret MacVicar Faculty Fellows: professor of brain and cognitive sciences John Gabrieli, associate professor of literature Marah Gubar, professor of biology Adam C. Martin, and associate professor of architecture Lawrence “Larry” Sass.

For more than 30 years, the MacVicar Faculty Fellows Program has recognized exemplary and sustained contributions to undergraduate education at MIT. The program is named in honor of Margaret MacVicar, the first dean for undergraduate education and founder of the Undergraduate Research Opportunities Program (UROP). New fellows are chosen every year through a competitive nomination process that includes submission of letters of support from colleagues, students, and alumni; review by an advisory committee led by the vice chancellor; and a final selection by the provost. Fellows are appointed to a 10-year term and receive $10,000 per year of discretionary funds.

Gabrieli, Gubar, Martin, and Sass join an elite group of more than 130 scholars from across the Institute who are committed to curricular innovation, excellence in teaching, and supporting students both in and out of the classroom.

John Gabrieli

“When I learned of this wonderful honor, I felt gratitude — for how MIT values teaching and learning, how my faculty colleagues bring such passion to their teaching, and how the students have such great curiosity for learning,” says new MacVicar Fellow John Gabrieli.

Gabrieli PhD ’87 received a bachelor’s degree in English from Yale University and his PhD in behavioral neuroscience from MIT. He is the Grover M. Hermann Professor in the Department of Brain and Cognitive sciences. Gabrieli is also an investigator in the McGovern Institute for Brain Research and the founding director of the MIT Integrated Learning Initiative (MITili). He holds appointments in the Department of Psychiatry at Massachusetts General Hospital and the Harvard Graduate School of Education, and studies the organization of memory, thought, and emotion in the human brain.

He joined Course 9 as a professor in 2005 and since then, he has taught over 3,000 undergraduates through the department’s introductory course, 9.00 (Introduction to Psychological Science). Gabrieli was recognized with departmental awards for excellence in teaching in 2009, 2012, and 2015. Highly sought after by undergraduate researchers, the Gabrieli Laboratory (GabLab) hosts five to 10 UROPs each year.

A unique element of Gabrieli’s classes is his passionate, hands-on teaching style and his use of interactive demonstrations, such as optical illusions and personality tests, to help students grasp some of the most fundamental topics in psychology.

His former teaching assistant Daniel Montgomery ’22 writes, “I was impressed by his enthusiasm and ability to keep students engaged throughout the lectures … John clearly has a desire to help students become excited about the material he’s teaching.”

Senior Elizabeth Carbonell agrees: “The excitement professor Gabrieli brought to lectures by starting with music every time made the classroom an enjoyable atmosphere conducive to learning … he always found a way to make every lecture relatable to the students, teaching psychological concepts that would shine a light on our own human emotions.”

Lecturer and 9.00 course coordinator Laura Frawley says, “John constantly innovates … He uses research-based learning techniques in his class, including blended learning, active learning, and retrieval practice.” His findings on blended learning resulted in two MITx offerings including 9.00x (Learning and Memory), which utilizes a nontraditional approach to assignments and exams to improve how students retrieve and remember information.

In addition, he is known for being a devoted teacher who believes in caring for the student as a whole. Through MITili’s Mental Wellness Initiative, Gabrieli, along with a compassionate team of faculty and staff, are working to better understand how mental health conditions impact learning.

Associate department head and associate professor of brain and cognitive sciences Josh McDermott calls him “an exceptional educator who has left his mark on generations of MIT undergraduate students with his captivating, innovative, and thoughtful approach to teaching.”

Mariana Gomez de Campo ’20 concurs: “There are certain professors that make their mark on students’ lives; professor Gabrieli permanently altered the course of mine.”

Laura Schulz, MacVicar Fellow and associate department head of brain and cognitive sciences, remarks, “His approach is visionary … John’s manner with students is unfailingly gracious … he hastens to remind them that they are as good as it gets, the smartest and brightest of their generation … it is the kind of warm, welcoming, inclusive approach to teaching that subtly but effectively reminds students that they belong here at MIT … It is little wonder that they love him.”

Marah Gubar

Marah Gubar joined MIT as an associate professor of literature in 2014. She received her BA in English literature from the University of Michigan at Ann Arbor and a PhD from Princeton University. Gubar taught in the English department at the University of Pittsburgh and served as director of the Children’s Literature Program. She received MIT’s James A. and Ruth Levitan Teaching Award in 2019 and the Teaching with Digital Technology Award in 2020.

Gubar’s research focuses on children’s literature, history of children’s theater, performance, and 19th- and 20th-century representations of childhood. Her research and pedagogies underscore the importance of integrated learning.

Colleagues at MIT note her efficacy in introducing new concepts and new subjects into the literature curriculum during her tenure as curricular chair. Gubar set the stage for wide-ranging curricular improvements, resulting in a host of literature subjects on interrelated topics within and across disciplines.

Gubar teaches several classes, including 21L.452 (Literature and Philosophy) and 21L.500 (How We Got to Hamilton). Her lectures provide uniquely enriching learning experiences in which her students are encouraged to dive into literary texts; craft thoughtful, persuasive arguments; and engage in lively intellectual debate.

Gubar encourages others to bring fresh ideas and think outside the box. For example, her seminar on “Hamilton” challenges students to recontextualize the hip-hop musical in several intellectual traditions. Professor Eric Klopfer, head of the Comparative Media Studies Program/Writing and interim head of literature, calls Gubar “a thoughtful, caring instructor, and course designer … She thinks critically about whose story is being told and by whom.”

MacVicar Fellow and professor of literature Stephen Tapscott praises her experimentation, abstract thinking, and storytelling: “Professor Gubar’s ability to frame intellectual questions in terms of problems, developments, and performance is an important dimension of the genius of her teaching.”

“Marah is hands-down the most enthusiastic, effective, and engaged professor I had the pleasure of learning from at MIT,” writes one student. “She’s one of the few instructors I’ve had who never feels the need to reassert her place in the didactic hierarchy, but approaches her students as intellectual equals.”

Tapscott continues, “She welcomes participation in ways that enrich the conversation, open new modes of communication, and empower students as autonomous literary critics. In professor Gubar’s classroom we learn by doing … and that progress also includes ‘doing’ textual analysis, cultural history, and abstract literary theory.”

Gubar is also a committed mentor and student testimonials highlight her supportive approach. One of her former students remarked that Gubar “has a strong drive to be inclusive, and truly cares about ‘getting it right’ … her passion for literature and teaching, together with her drive for inclusivity, her ability to take accountability, and her compassion and empathy for her students, make [her] a truly remarkable teacher.”

On receiving this award Marah Gubar writes, “The best word I can think of to describe how I reacted to hearing that I had received this very overwhelming honor is ‘plotzing.’ The Yiddish verb ‘to plotz’ literally means to crack, burst, or collapse, so that captures how undone I was. I started to cry, because it suddenly struck me how much joy my father, Edward Gubar, would have taken in this amazing news. He was a teacher, too, and he died during the first phase of this terrible pandemic that we’re still struggling to get through.”

Adam C. Martin

Adam C. Martin is a professor and undergraduate officer in the Department of Biology. He studies the molecular mechanisms that underlie tissue form and function. His research interests include gastrulation, embryotic development, cytoskeletal dynamics, and the coordination of cellular behavior. Martin received his PhD from the University of California at Berkeley and his BS in biology (genetics) from Cornell University. Martin joined the Course 7 faculty in 2011.

“I am overwhelmed with gratitude knowing that this has come from our students. The fact that they spent time to contribute to a nomination is incredibly meaningful to me,” says Martin. “I want to also thank all of my faculty colleagues with whom I have taught, appreciate, and learned immensely from over the past 12 years. I am a better teacher because of them and inspired by their dedication.”

He is committed to undergraduate education, teaching several key department offerings including 7.06 (Cell Biology), 7.016 (Introductory Biology), 7.002 (Fundamentals of Experimental Molecular Biology), and 7.102 (Introduction to Molecular Biology Techniques).

Martin’s style combines academic and scientific expertise with creative elements like props and demonstrations. His “energy and passion for the material” is obvious, writes Iain Cheeseman, associate department head and the Herman and Margaret Sokol Professor of Biology. “In addition to creating engaging lectures, Adam went beyond the standard classroom requirements to develop videos and animations (in collaboration with the Biology MITx team) to illustrate core cell biological approaches and concepts.”

What sets Martin apart is his connection with students, his positive spirit, and his welcoming demeanor. Apolonia Gardner ’22 reflects on the way he helped her outside of class through his running group, which connects younger students with seniors in his lab. “Professor Martin was literally committed to ‘going the extra mile’ by inviting his students to join him on runs around the Charles River on Friday afternoons,” she says.

Amy Keating, department head and Jay A. Stein professor of biology, and professor of biological engineering, goes on to praise Martin’s ability to attract students to Course 7 and guide them through their educational experience in his role as the director of undergraduate studies. “He hosts social events, presides at our undergraduate research symposium and the department’s undergraduate graduation and awards banquet, and works with the Biology Undergraduate Student Association,” she says.

As undergraduate officer, Martin is involved in both advising and curriculum building. He mentors UROP students, serves as a first-year advisor, and is a current member of MIT’s Committee on the Undergraduate Program (CUP).

Martin also brings a commitment to diversity, equity, and inclusion (DEI) as evidenced by his creation of a DEI journal club in his lab so that students have a dedicated space to discuss issues and challenges. Course 7 DEI officer Hallie Dowling-Huppert writes that Martin “thinks deeply about how DEI efforts are created to ensure that department members receive the maximum benefit. Adam considers all perspectives when making decisions, and is extremely empathetic and caring towards his students.”

“He makes our world so much better,” Keating observes. “Adam is a gem.”

Lawrence “Larry” Sass

Larry Sass SM ’94, PhD ’00 is an associate professor in the Department of Architecture. He earned his PhD and SM in architecture at MIT, and has a BArch from Pratt Institute in New York City. Sass joined the faculty in the Department of Architecture in 2002. His work focuses on the delivery of affordable housing for low-income families. He was included in an exhibit titled “Home Delivery: Fabricating the Modern Dwelling” at the Museum of Modern Art in New York City.

Sass’s teaching blends computation with design. His two signature courses, 4.500 (Design Computation: Art, Objects and Space) and 4.501 (Tiny Fab: Advancements in Rapid Design and Fabrication of Small Homes), reflect his specialization in digitally fabricating buildings and furniture from machines.

Professor and head of architecture Nicholas de Monchaux writes, “his classes provide crucial instruction and practice with 3D modeling and computer-generated rendering and animation …  [He] links digital design to fabrication, in a process that invites students to define desirable design attributes of an object, develop a digital model, prototype it, and construct it at full scale.”

More generally, Sass’ approach is to help students build confidence in their own design process through hands-on projects. MIT Class of 1942 Professor John Ochsendorf, MacVicar Fellow, and founding director of the Morningside Academy for Design with appointments in the departments of architecture and civil and environmental engineering, confirms, “Larry’s teaching is a perfect embodiment of the ‘mens et manus’ spirit … [he] requires his students to go back and forth from mind and hand throughout each design project.”

Students say that his classes are a journey of self-discovery, allowing them to learn more about themselves and their own abilities. Senior Natasha Hirt notes, “What I learned from Larry was not something one can glean from a textbook, but a new way of seeing space … he tectonically shifted my perspective on buildings. He also shifted my perspective on myself. I’m a better designer for his teachings, and perhaps more importantly, I better understand how I design.”

Senior Izzi Waitz echoes this sentiment: “Larry emphasizes the importance of intentionally thinking through your designs and being confident in your choices … he challenges, questions, and prompts you so that you learn to defend and support yourself on your own.”

As a UROP coordinator, Sass assures students that the “sky is the limit” and all ideas are welcome. Postgraduate teaching fellow and research associate Myles Sampson says, “During the last year of my SM program, I assisted Larry in conducting a year-long UROP project … He structured the learning experience in a way that allowed the students to freely flex their design muscles: no idea was too outrageous.”

Sass is equally devoted to his students outside the classroom. In his role as head of house at MacGregor House, he lives in community with more than 300 undergraduates each year, providing academic guidance, creating residential programs and recreational activities, and ensuring that student wellness and mental health is a No. 1 priority.

Professor of architecture and MacVicar Fellow Les Norford says, “In two significant ways, Larry has been ahead of his time: combining digital representation and design with making and being alert to the well-being of his students.”

“In his kindness, he honors the memory of Margaret MacVicar, as well as the spirit of MIT itself,” Hirt concludes. “He is a designer, a craftsman, and an innovator. He is an inspiration and a compass.”

On receiving this award, Sass is full of excitement: “I love teaching and being part of the MIT community. I am grateful for the opportunity to be part of the MacVicar family of fellows.”

New insights into training dynamics of deep classifiers

by MIT Center for Brains, Minds and Machines |

A new study from researchers at MIT and Brown University characterizes several properties that emerge during the training of deep classifiers, a type of artificial neural network commonly used for classification tasks such as image classification, speech recognition, and natural language processing.

The paper, “Dynamics in Deep Classifiers trained with the Square Loss: Normalization, Low Rank, Neural Collapse and Generalization Bounds,” published today in the journal Research, is the first of its kind to theoretically explore the dynamics of training deep classifiers with the square loss and how properties such as rank minimization, neural collapse, and dualities between the activation of neurons and the weights of the layers are intertwined.

In the study, the authors focused on two types of deep classifiers: fully connected deep networks and convolutional neural networks (CNNs).

A previous study examined the structural properties that develop in large neural networks at the final stages of training. That study focused on the last layer of the network and found that deep networks trained to fit a training dataset will eventually reach a state known as “neural collapse.” When neural collapse occurs, the network maps multiple examples of a particular class (such as images of cats) to a single template of that class. Ideally, the templates for each class should be as far apart from each other as possible, allowing the network to accurately classify new examples.

An MIT group based at the MIT Center for Brains, Minds and Machines studied the conditions under which networks can achieve neural collapse. Deep networks that have the three ingredients of stochastic gradient descent (SGD), weight decay regularization (WD), and weight normalization (WN) will display neural collapse if they are trained to fit their training data. The MIT group has taken a theoretical approach — as compared to the empirical approach of the earlier study — proving that neural collapse emerges from the minimization of the square loss using SGD, WD, and WN.

Co-author and MIT McGovern Institute postdoc Akshay Rangamani states, “Our analysis shows that neural collapse emerges from the minimization of the square loss with highly expressive deep neural networks. It also highlights the key roles played by weight decay regularization and stochastic gradient descent in driving solutions towards neural collapse.”

Weight decay is a regularization technique that prevents the network from over-fitting the training data by reducing the magnitude of the weights. Weight normalization scales the weight matrices of a network so that they have a similar scale. Low rank refers to a property of a matrix where it has a small number of non-zero singular values. Generalization bounds offer guarantees about the ability of a network to accurately predict new examples that it has not seen during training.

The authors found that the same theoretical observation that predicts a low-rank bias also predicts the existence of an intrinsic SGD noise in the weight matrices and in the output of the network. This noise is not generated by the randomness of the SGD algorithm but by an interesting dynamic trade-off between rank minimization and fitting of the data, which provides an intrinsic source of noise similar to what happens in dynamic systems in the chaotic regime. Such a random-like search may be beneficial for generalization because it may prevent over-fitting.

“Interestingly, this result validates the classical theory of generalization showing that traditional bounds are meaningful. It also provides a theoretical explanation for the superior performance in many tasks of sparse networks, such as CNNs, with respect to dense networks,” comments co-author and MIT McGovern Institute postdoc Tomer Galanti. In fact, the authors prove new norm-based generalization bounds for CNNs with localized kernels, that is a network with sparse connectivity in their weight matrices.

In this case, generalization can be orders of magnitude better than densely connected networks. This result validates the classical theory of generalization, showing that its bounds are meaningful, and goes against a number of recent papers expressing doubts about past approaches to generalization. It also provides a theoretical explanation for the superior performance of sparse networks, such as CNNs, with respect to dense networks. Thus far, the fact that CNNs and not dense networks represent the success story of deep networks has been almost completely ignored by machine learning theory. Instead, the theory presented here suggests that this is an important insight in why deep networks work as well as they do.

“This study provides one of the first theoretical analyses covering optimization, generalization, and approximation in deep networks and offers new insights into the properties that emerge during training,” says co-author Tomaso Poggio, the Eugene McDermott Professor at the Department of Brain and Cognitive Sciences at MIT and co-director of the Center for Brains, Minds and Machines. “Our results have the potential to advance our understanding of why deep learning works as well as it does.”

School of Science presents 2023 Infinite Expansion Awards

Anne Trafton |

The MIT School of Science has announced seven postdocs and research scientists as recipients of the 2023 Infinite Expansion Award. Nominated by their peers and mentors, the awardees are recognized not only for their exceptional science, but for mentoring and advising junior colleagues, supporting educational programs, working with the MIT Postdoctoral Association, or contributing some other way to the Institute.

The 2023 Infinite Expansion award winners in the School of Science are:

  • Kyle Jenks, a postdoc in the Picower Institute for Learning and Memory, nominated by professor and Picower Institute investigator Mriganka Sur;
  • Matheus Victor, a postdoc in the Picower Institute, nominated by professor and Picower Institute director Li-Huei Tsai.

A monetary award is granted to recipients, and a celebratory reception will be held for the winners this spring with family, friends, nominators, and recipients of the Infinite Expansion Award.

How Huntington’s disease affects different neurons

Anne Trafton |

In patients with Huntington’s disease, neurons in a part of the brain called the striatum are among the hardest-hit. Degeneration of these neurons contributes to patients’ loss of motor control, which is one of the major hallmarks of the disease.

Neuroscientists at MIT have now shown that two distinct cell populations in the striatum are affected differently by Huntington’s disease. They believe that neurodegeneration of one of these populations leads to motor impairments, while damage to the other population, located in structures called striosomes, may account for the mood disorders that are often see in the early stages of the disease.

“As many as 10 years ahead of the motor diagnosis, Huntington’s patients can experience mood disorders, and one possibility is that the striosomes might be involved in these,” says Ann Graybiel, an MIT Institute Professor, a member of MIT’s McGovern Institute for Brain Research, and one of the senior authors of the study.

Using single-cell RNA sequencing to analyze the genes expressed in mouse models of Huntington’s disease and postmortem brain samples from Huntington’s patients, the researchers found that cells of the striosomes and another structure, the matrix, begin to lose their distinguishing features as the disease progresses. The researchers hope that their mapping of the striatum and how it is affected by Huntington’s could help lead to new treatments that target specific cells within the brain.

This kind of analysis could also shed light on other brain disorders that affect the striatum, such as Parkinson’s disease and autism spectrum disorder, the researchers say.

Myriam Heiman, an associate professor in MIT’s Department of Brain and Cognitive Sciences and a member of the Picower Institute for Learning and Memory, and Manolis Kellis, a professor of computer science in MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) and a member of the Broad Institute of MIT and Harvard, are also senior authors of the study. Ayano Matsushima, a McGovern Institute research scientist, and Sergio Sebastian Pineda, an MIT graduate student, are the lead authors of the paper, which appears in Nature Communications.

Neuron vulnerability

Huntington’s disease leads to degeneration of brain structures called the basal ganglia, which are responsible for control of movement and also play roles in other behaviors, as well as emotions. For many years, Graybiel has been studying the striatum, a part of the basal ganglia that is involved in making decisions that require evaluating the outcomes of a particular action.

Many years ago, Graybiel discovered that the striatum is divided into striosomes, which are clusters of neurons, and the matrix, which surrounds the striosomes. She has also shown that striosomes are necessary for making decisions that require an anxiety-provoking cost-benefit analysis.

In a 2007 study, Richard Faull of the University of Auckland discovered that in postmortem brain tissue from Huntington’s patients, the striosomes showed a great deal of degeneration. Faull also found that while those patients were alive, many of them had shown signs of mood disorders such as depression before their motor symptoms developed.

To further explore the connections between the striatum and the mood and motor effects of Huntington’s, Graybiel teamed up with Kellis and Heiman to study the gene expression patterns of striosomal and matrix cells. To do that, the researchers used single-cell RNA sequencing to analyze human brain samples and brain tissue from two mouse models of Huntington’s disease.

Within the striatum, neurons can be classified as either D1 or D2 neurons. D1 neurons are involved in the “go” pathway, which initiates an action, and D2 neurons are part of the “no-go” pathway, which suppresses an action. D1 and D2 neurons can both be found within either the striosomes and the matrix.

The analysis of RNA expression in each of these types of cells revealed that striosomal neurons are harder hit by Huntington’s than matrix neurons. Furthermore, within the striosomes, D2 neurons are more vulnerable than D1.

The researchers also found that these four major cell types begin to lose their identifying molecular identities and become more difficult to distinguish from one another in Huntington’s disease. “Overall, the distinction between striosomes and matrix becomes really blurry,” Graybiel says.

Striosomal disorders

The findings suggest that damage to the striosomes, which are known to be involved in regulating mood, may be responsible for the mood disorders that strike Huntington’s patients in the early stages of the disease. Later on, degeneration of the matrix neurons likely contributes to the decline of motor function, the researchers say.

In future work, the researchers hope to explore how degeneration or abnormal gene expression in the striosomes may contribute to other brain disorders.

Previous research has shown that overactivity of striosomes can lead to the development of repetitive behaviors such as those seen in autism, obsessive compulsive disorder, and Tourette’s syndrome. In this study, at least one of the genes that the researchers discovered was overexpressed in the striosomes of Huntington’s brains is also linked to autism.

Additionally, many striosome neurons project to the part of the brain that is most affected by Parkinson’s disease (the substantia nigra, which produces most of the brain’s dopamine).

“There are many, many disorders that probably involve the striatum, and now, partly through transcriptomics, we’re working to understand how all of this could fit together,” Graybiel says.

The research was funded by the Saks Kavanaugh Foundation, the CHDI Foundation, the National Institutes of Health, the Nancy Lurie Marks Family Foundation, the Simons Foundation, the JPB Foundation, the Kristin R. Pressman and Jessica J. Pourian ’13 Fund, and Robert Buxton.

Self-assembling proteins can store cellular “memories”

Anne Trafton |

As cells perform their everyday functions, they turn on a variety of genes and cellular pathways. MIT engineers have now coaxed cells to inscribe the history of these events in a long protein chain that can be imaged using a light microscope.

Cells programmed to produce these chains continuously add building blocks that encode particular cellular events. Later, the ordered protein chains can be labeled with fluorescent molecules and read under a microscope, allowing researchers to reconstruct the timing of the events.

This technique could help shed light on the steps that underlie processes such as memory formation, response to drug treatment, and gene expression.

“There are a lot of changes that happen at organ or body scale, over hours to weeks, which cannot be tracked over time,” says Edward Boyden, the Y. Eva Tan Professor in Neurotechnology, a professor of biological engineering and brain and cognitive sciences at MIT, a Howard Hughes Medical Institute investigator, and a member of MIT’s McGovern Institute for Brain Research and Koch Institute for Integrative Cancer Research.

If the technique could be extended to work over longer time periods, it could also be used to study processes such as aging and disease progression, the researchers say.

Boyden is the senior author of the study, which appears today in Nature Biotechnology. Changyang Linghu, a former J. Douglas Tan Postdoctoral Fellow at the McGovern Institute, who is now an assistant professor at the University of Michigan, is the lead author of the paper.

Cellular history

Biological systems such as organs contain many different kinds of cells, all of which have distinctive functions. One way to study these functions is to image proteins, RNA, or other molecules inside the cells, which provide hints to what the cells are doing. However, most methods for doing this offer only a glimpse of a single moment in time, or don’t work well with very large populations of cells.

“Biological systems are often composed of a large number of different types of cells. For example, the human brain has 86 billion cells,” Linghu says. “To understand those kinds of biological systems, we need to observe physiological events over time in these large cell populations.”

To achieve that, the research team came up with the idea of recording cellular events as a series of protein subunits that are continuously added to a chain. To create their chains, the researchers used engineered protein subunits, not normally found in living cells, that can self-assemble into long filaments.

The researchers designed a genetically encoded system in which one of these subunits is continuously produced inside cells, while the other is generated only when a specific event occurs. Each subunit also contains a very short peptide called an epitope tag — in this case, the researchers chose tags called HA and V5. Each of these tags can bind to a different fluorescent antibody, making it easy to visualize the tags later on and determine the sequence of the protein subunits.

For this study, the researchers made production of the V5-containing subunit contingent on the activation of a gene called c-fos, which is involved in encoding new memories. HA-tagged subunits make up most of the chain, but whenever the V5 tag shows up in the chain, that means that c-fos was activated during that time.

“We’re hoping to use this kind of protein self-assembly to record activity in every single cell,” Linghu says. “It’s not only a snapshot in time, but also records past history, just like how tree rings can permanently store information over time as the wood grows.”

Recording events

In this study, the researchers first used their system to record activation of c-fos in neurons growing in a lab dish. The c-fos gene was activated by chemically induced activation of the neurons, which caused the V5 subunit to be added to the protein chain.

To explore whether this approach could work in the brains of animals, the researchers programmed brain cells of mice to generate protein chains that would reveal when the animals were exposed to a particular drug. Later, the researchers were able to detect that exposure by preserving the tissue and analyzing it with a light microscope.

The researchers designed their system to be modular, so that different epitope tags can be swapped in, or different types of cellular events can be detected, including, in principle, cell division or activation of enzymes called protein kinases, which help control many cellular pathways.

The researchers also hope to extend the recording period that they can achieve. In this study, they recorded events for several days before imaging the tissue. There is a tradeoff between the amount of time that can be recorded and the time resolution, or frequency of event recording, because the length of the protein chain is limited by the size of the cell.

“The total amount of information it could store is fixed, but we could in principle slow down or increase the speed of the growth of the chain,” Linghu says. “If we want to record for a longer time, we could slow down the synthesis so that it will reach the size of the cell within, let’s say two weeks. In that way we could record longer, but with less time resolution.”

The researchers are also working on engineering the system so that it can record multiple types of events in the same chain, by increasing the number of different subunits that can be incorporated.

The research was funded by the Hock E. Tan and K. Lisa Yang Center for Autism Research, John Doerr, the National Institutes of Health, the National Science Foundation, the U.S. Army Research Office, and the Howard Hughes Medical Institute.

New sensor uses MRI to detect light deep in the brain

Anne Trafton |

Using a specialized MRI sensor, MIT researchers have shown that they can detect light deep within tissues such as the brain.

Imaging light in deep tissues is extremely difficult because as light travels into tissue, much of it is either absorbed or scattered. The MIT team overcame that obstacle by designing a sensor that converts light into a magnetic signal that can be detected by MRI (magnetic resonance imaging).

This type of sensor could be used to map light emitted by optical fibers implanted in the brain, such as the fibers used to stimulate neurons during optogenetic experiments. With further development, it could also prove useful for monitoring patients who receive light-based therapies for cancer, the researchers say.

“We can image the distribution of light in tissue, and that’s important because people who use light to stimulate tissue or to measure from tissue often don’t quite know where the light is going, where they’re stimulating, or where the light is coming from. Our tool can be used to address those unknowns,” says Alan Jasanoff, an MIT professor of biological engineering, brain and cognitive sciences, and nuclear science and engineering.

Jasanoff, who is also an associate investigator at MIT’s McGovern Institute for Brain Research, is the senior author of the study, which appears today in Nature Biomedical Engineering. Jacob Simon PhD ’21 and MIT postdoc Miriam Schwalm are the paper’s lead authors, and Johannes Morstein and Dirk Trauner of New York University are also authors of the paper.

A light-sensitive probe

Scientists have been using light to study living cells for hundreds of years, dating back to the late 1500s, when the light microscope was invented. This kind of microscopy allows researchers to peer inside cells and thin slices of tissue, but not deep inside an organism.

“One of the persistent problems in using light, especially in the life sciences, is that it doesn’t do a very good job penetrating many materials,” Jasanoff says. “Biological materials absorb light and scatter light, and the combination of those things prevents us from using most types of optical imaging for anything that involves focusing in deep tissue.”

To overcome that limitation, Jasanoff and his students decided to design a sensor that could transform light into a magnetic signal.

“We wanted to create a magnetic sensor that responds to light locally, and therefore is not subject to absorbance or scattering. Then this light detector can be imaged using MRI,” he says.

Jasanoff’s lab has previously developed MRI probes that can interact with a variety of molecules in the brain, including dopamine and calcium. When these probes bind to their targets, it affects the sensors’ magnetic interactions with the surrounding tissue, dimming or brightening the MRI signal.

To make a light-sensitive MRI probe, the researchers decided to encase magnetic particles in a nanoparticle called a liposome. The liposomes used in this study are made from specialized light-sensitive lipids that Trauner had previously developed. When these lipids are exposed to a certain wavelength of light, the liposomes become more permeable to water, or “leaky.” This allows the magnetic particles inside to interact with water and generate a signal detectable by MRI.

The particles, which the researchers called liposomal nanoparticle reporters (LisNR), can switch from permeable to impermeable depending on the type of light they’re exposed to. In this study, the researchers created particles that become leaky when exposed to ultraviolet light, and then become impermeable again when exposed to blue light. The researchers also showed that the particles could respond to other wavelengths of light.

“This paper shows a novel sensor to enable photon detection with MRI through the brain. This illuminating work introduces a new avenue to bridge photon and proton-driven neuroimaging studies,” says Xin Yu, an assistant professor radiology at Harvard Medical School, who was not involved in the study.

Mapping light

The researchers tested the sensors in the brains of rats — specifically, in a part of the brain called the striatum, which is involved in planning movement and responding to reward. After injecting the particles throughout the striatum, the researchers were able to map the distribution of light from an optical fiber implanted nearby.

The fiber they used is similar to those used for optogenetic stimulation, so this kind of sensing could be useful to researchers who perform optogenetic experiments in the brain, Jasanoff says.

“We don’t expect that everybody doing optogenetics will use this for every experiment — it’s more something that you would do once in a while, to see whether a paradigm that you’re using is really producing the profile of light that you think it should be,” Jasanoff says.

In the future, this type of sensor could also be useful for monitoring patients receiving treatments that involve light, such as photodynamic therapy, which uses light from a laser or LED to kill cancer cells.

The researchers are now working on similar probes that could be used to detect light emitted by luciferases, a family of glowing proteins that are often used in biological experiments. These proteins can be used to reveal whether a particular gene is activated or not, but currently they can only be imaged in superficial tissue or cells grown in a lab dish.

Jasanoff also hopes to use the strategy used for the LisNR sensor to design MRI probes that can detect stimuli other than light, such as neurochemicals or other molecules found in the brain.

“We think that the principle that we use to construct these sensors is quite broad and can be used for other purposes too,” he says.

The research was funded by the National Institutes of Health, the G. Harold and Leila Y. Mathers Foundation, a Friends of the McGovern Fellowship from the McGovern Institute for Brain Research, the MIT Neurobiological Engineering Training Program, and a Marie Curie Individual Fellowship from the European Commission.

This is your brain. This is your brain on code

by Steve Nadis | MIT CSAIL |

Functional magnetic resonance imaging (fMRI), which measures changes in blood flow throughout the brain, has been used over the past couple of decades for a variety of applications, including “functional anatomy” — a way of determining which brain areas are switched on when a person carries out a particular task. fMRI has been used to look at people’s brains while they’re doing all sorts of things — working out math problems, learning foreign languages, playing chess, improvising on the piano, doing crossword puzzles, and even watching TV shows like “Curb Your Enthusiasm.”

One pursuit that’s received little attention is computer programming — both the chore of writing code and the equally confounding task of trying to understand a piece of already-written code. “Given the importance that computer programs have assumed in our everyday lives,” says Shashank Srikant, a PhD student in MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL), “that’s surely worth looking into. So many people are dealing with code these days — reading, writing, designing, debugging — but no one really knows what’s going on in their heads when that happens.” Fortunately, he has made some “headway” in that direction in a paper — written with MIT colleagues Benjamin Lipkin (the paper’s other lead author, along with Srikant), Anna Ivanova, Evelina Fedorenko, and Una-May O’Reilly — that was presented earlier this month at the Neural Information Processing Systems Conference held in New Orleans.

The new paper built on a 2020 study, written by many of the same authors, which used fMRI to monitor the brains of programmers as they “comprehended” small pieces, or snippets, of code. (Comprehension, in this case, means looking at a snippet and correctly determining the result of the computation performed by the snippet.) The 2020 work showed that code comprehension did not consistently activate the language system, brain regions that handle language processing, explains Fedorenko, a brain and cognitive sciences (BCS) professor and a coauthor of the earlier study. “Instead, the multiple demand network — a brain system that is linked to general reasoning and supports domains like mathematical and logical thinking — was strongly active.” The current work, which also utilizes MRI scans of programmers, takes “a deeper dive,” she says, seeking to obtain more fine-grained information.

Whereas the previous study looked at 20 to 30 people to determine which brain systems, on average, are relied upon to comprehend code, the new research looks at the brain activity of individual programmers as they process specific elements of a computer program. Suppose, for instance, that there’s a one-line piece of code that involves word manipulation and a separate piece of code that entails a mathematical operation. “Can I go from the activity we see in the brains, the actual brain signals, to try to reverse-engineer and figure out what, specifically, the programmer was looking at?” Srikant asks. “This would reveal what information pertaining to programs is uniquely encoded in our brains.” To neuroscientists, he notes, a physical property is considered “encoded” if they can infer that property by looking at someone’s brain signals.

Take, for instance, a loop — an instruction within a program to repeat a specific operation until the desired result is achieved — or a branch, a different type of programming instruction than can cause the computer to switch from one operation to another. Based on the patterns of brain activity that were observed, the group could tell whether someone was evaluating a piece of code involving a loop or a branch. The researchers could also tell whether the code related to words or mathematical symbols, and whether someone was reading actual code or merely a written description of that code.

That addressed a first question that an investigator might ask as to whether something is, in fact, encoded. If the answer is yes, the next question might be: where is it encoded? In the above-cited cases — loops or branches, words or math, code or a description thereof — brain activation levels were found to be comparable in both the language system and the multiple demand network.

A noticeable difference was observed, however, when it came to code properties related to what’s called dynamic analysis.

Programs can have “static” properties — such as the number of numerals in a sequence — that do not change over time. “But programs can also have a dynamic aspect, such as the number of times a loop runs,” Srikant says. “I can’t always read a piece of code and know, in advance, what the run time of that program will be.” The MIT researchers found that for dynamic analysis, information is encoded much better in the multiple demand network than it is in the language processing center. That finding was one clue in their quest to see how code comprehension is distributed throughout the brain — which parts are involved and which ones assume a bigger role in certain aspects of that task.

The team carried out a second set of experiments, which incorporated machine learning models called neural networks that were specifically trained on computer programs. These models have been successful, in recent years, in helping programmers complete pieces of code. What the group wanted to find out was whether the brain signals seen in their study when participants were examining pieces of code resembled the patterns of activation observed when neural networks analyzed the same piece of code. And the answer they arrived at was a qualified yes.

“If you put a piece of code into the neural network, it produces a list of numbers that tells you, in some way, what the program is all about,” Srikant says. Brain scans of people studying computer programs similarly produce a list of numbers. When a program is dominated by branching, for example, “you see a distinct pattern of brain activity,” he adds, “and you see a similar pattern when the machine learning model tries to understand that same snippet.”

Mariya Toneva of the Max Planck Institute for Software Systems considers findings like this “particularly exciting. They raise the possibility of using computational models of code to better understand what happens in our brains as we read programs,” she says.

The MIT scientists are definitely intrigued by the connections they’ve uncovered, which shed light on how discrete pieces of computer programs are encoded in the brain. But they don’t yet know what these recently-gleaned insights can tell us about how people carry out more elaborate plans in the real world. Completing tasks of this sort — such as going to the movies, which requires checking showtimes, arranging for transportation, purchasing tickets, and so forth — could not be handled by a single unit of code and just a single algorithm. Successful execution of such a plan would instead require “composition” — stringing together various snippets and algorithms into a sensible sequence that leads to something new, just like assembling individual bars of music in order to make a song or even a symphony. Creating models of code composition, says O’Reilly, a principal research scientist at CSAIL, “is beyond our grasp at the moment.”

Lipkin, a BCS PhD student, considers this the next logical step — figuring out how to “combine simple operations to build complex programs and use those strategies to effectively address general reasoning tasks.” He further believes that some of the progress toward that goal achieved by the team so far owes to its interdisciplinary makeup. “We were able to draw from individual experiences with program analysis and neural signal processing, as well as combined work on machine learning and natural language processing,” Lipkin says. “These types of collaborations are becoming increasingly common as neuro- and computer scientists join forces on the quest towards understanding and building general intelligence.”

This project was funded by grants from the MIT-IBM Watson AI lab, MIT Quest Initiative, National Science Foundation, National Institutes of Health, McGovern Institute of Brain Research, MIT Department of Brain and Cognitive Sciences, and the Simons Center for the Social Brain.