Researcher: Mehrdad Jazayeri

Study reveals how the brain tracks objects in motion

Anne Trafton |

Catching a bouncing ball or hitting a ball with a racket requires estimating when the ball will arrive. Neuroscientists have long thought that the brain does this by calculating the speed of the moving object. However, a new study from MIT shows that the brain’s approach is more complex.

The new findings suggest that in addition to tracking speed, the brain incorporates information about the rhythmic patterns of an object’s movement: for example, how long it takes a ball to complete one bounce. In their new study, the researchers found that people make much more accurate estimates when they have access to information about both the speed of a moving object and the timing of its rhythmic patterns.

“People get really good at this when they have both types of information available,” says Mehrdad Jazayeri, the Robert A. Swanson Career Development Professor of Life Sciences and a member of MIT’s McGovern Institute for Brain Research. “It’s like having input from multiple senses. The statistical knowledge that we have about the world we’re interacting with is richer when we use multiple senses.”

Jazayeri is the senior author of the study, which appears in the Proceedings of the National Academy of Sciences the week of March 5. The paper’s lead author is MIT graduate student Chia-Jung Chang.

Objects in motion

Much of the information we process about objects moving around us comes from visual tracking of the objects. Our brains can use information about an object’s speed and the distance it has to cover to calculate when it will reach a certain point. Jazayeri, who studies how the brain keeps time, was intrigued by the fact that much of the movement we see also has a rhythmic element, such as the bouncing of a ball.

“It occurred to us to ask, how can it be that the brain doesn’t use this information? It would seem very strange if all this richness of additional temporal structure is not part of the way we evaluate where things are around us and how things are going to happen,” Jazayeri says.

There are many other sensory processing tasks for which the brain uses multiple sources of input. For example, to interpret language, we use both the sound we hear and the movement of the speaker’s lips, if we can see them. When we touch an object, we estimate its size based on both what we see and what we feel with our fingers.

In the case of perceiving object motion, teasing out the role of rhythmic timing, as opposed to speed, can be difficult. “I can ask someone to do a task, but then how do I know if they’re using speed or they’re using time, if both of them are always available?” Jazayeri says.

To overcome that, the researchers devised a task in which they could control how much timing information was available. They measured performance in human volunteers as they performed the task.

During the task, the study participants watched a ball as it moved in a straight line. After traveling some distance, the ball went behind an obstacle, so the participants could no longer see it. They were asked to press a button at the time when they expected the ball to reappear.

Performance varied greatly depending on how much of the ball’s path was visible before it went behind the obstacle. If the participants saw the ball travel a very short distance before disappearing, they did not do well. As the distance before disappearance became longer, they were better able to calculate the ball’s speed, so their performance improved but eventually plateaued.

After that plateau, there was a significant jump in performance when the distance before disappearance grew until it was exactly the same as the width of the obstacle. In that case, when the path seen before disappearance was equal to the path the ball traveled behind the obstacle, the participants improved dramatically, because they knew that the time spent behind the obstacle would be the same as the time it took to reach the obstacle.

When the distance traveled to reach the obstacle became longer than the width of the obstacle, performance dropped again.

“It’s so important to have this extra information available, and when we have it, we use it,” Jazayeri says. “Temporal structure is so important that when you lose it, even at the expense of getting better visual information, people’s performance gets worse.”

Integrating information

The researchers also tested several computer models of how the brain performs this task, and found that the only model that could accurately replicate their experimental results was one in which the brain measures speed and timing in two different areas and then combines them.

Previous studies suggest that the brain performs timing estimates in premotor areas of the cortex, which plays a role in planning movement; speed, which usually requires visual input, is calculated in visual cortex. These inputs are likely combined in parts of the brain responsible for spatial attention and tracking objects in space, which occurs in the parietal cortex, Jazayeri says.

In future studies, Jazayeri hopes to measure brain activity in animals trained to perform the same task that human subjects did in this study. This could shed further light on where this processing takes place and could also reveal what happens in the brain when it makes incorrect estimates.

The research was funded by the McGovern Institute for Brain Research.

How the brain keeps time

Anne Trafton |

Timing is critical for playing a musical instrument, swinging a baseball bat, and many other activities. Neuroscientists have come up with several models of how the brain achieves its exquisite control over timing, the most prominent being that there is a centralized clock, or pacemaker, somewhere in the brain that keeps time for the entire brain.

However, a new study from MIT researchers provides evidence for an alternative timekeeping system that relies on the neurons responsible for producing a specific action. Depending on the time interval required, these neurons compress or stretch out the steps they take to generate the behavior at a specific time.

“What we found is that it’s a very active process. The brain is not passively waiting for a clock to reach a particular point,” says Mehrdad Jazayeri, the Robert A. Swanson Career Development Professor of Life Sciences, a member of MIT’s McGovern Institute for Brain Research, and the senior author of the study.

MIT postdoc Jing Wang and former postdoc Devika Narain are the lead authors of the paper, which appears in the Dec. 4 issue of Nature Neuroscience. Graduate student Eghbal Hosseini is also an author of the paper.

Flexible control

One of the earliest models of timing control, known as the clock accumulator model, suggested that the brain has an internal clock or pacemaker that keeps time for the rest of the brain. A later variation of this model suggested that instead of using a central pacemaker, the brain measures time by tracking the synchronization between different brain wave frequencies.

Although these clock models are intuitively appealing, Jazayeri says, “they don’t match well with what the brain does.”

No one has found evidence for a centralized clock, and Jazayeri and others wondered if parts of the brain that control behaviors that require precise timing might perform the timing function themselves. “People now question why would the brain want to spend the time and energy to generate a clock when it’s not always needed. For certain behaviors you need to do timing, so perhaps the parts of the brain that subserve these functions can also do timing,” he says.

To explore this possibility, the researchers recorded neuron activity from three brain regions in animals as they performed a task at two different time intervals — 850 milliseconds or 1,500 milliseconds.

The researchers found a complicated pattern of neural activity during these intervals. Some neurons fired faster, some fired slower, and some that had been oscillating began to oscillate faster or slower. However, the researchers’ key discovery was that no matter the neurons’ response, the rate at which they adjusted their activity depended on the time interval required.

At any point in time, a collection of neurons is in a particular “neural state,” which changes over time as each individual neuron alters its activity in a different way. To execute a particular behavior, the entire system must reach a defined end state. The researchers found that the neurons always traveled the same trajectory from their initial state to this end state, no matter the interval. The only thing that changed was the rate at which the neurons traveled this trajectory.

When the interval required was longer, this trajectory was “stretched,” meaning the neurons took more time to evolve to the final state. When the interval was shorter, the trajectory was compressed.

“What we found is that the brain doesn’t change the trajectory when the interval changes, it just changes the speed with which it goes from the initial internal state to the final state,” Jazayeri says.

Dean Buonomano, a professor of behavioral neuroscience at the University of California at Los Angeles, says that the study “provides beautiful evidence that timing is a distributed process in the brain — that is, there is no single master clock.”

“This work also supports the notion that the brain does not tell time using a clock-like mechanism, but rather relies on the dynamics inherent to neural circuits, and that as these dynamics increase and decrease in speed, animals move more quickly or slowly,” adds Buonomano, who was not involved in the research.

Neural networks

The researchers focused their study on a brain loop that connects three regions: the dorsomedial frontal cortex, the caudate, and the thalamus. They found this distinctive neural pattern in the dorsomedial frontal cortex, which is involved in many cognitive processes, and the caudate, which is involved in motor control, inhibition, and some types of learning. However, in the thalamus, which relays motor and sensory signals, they found a different pattern: Instead of altering the speed of their trajectory, many of the neurons simply increased or decreased their firing rate, depending on the interval required.

Jazayeri says this finding is consistent with the possibility that the thalamus is instructing the cortex on how to adjust its activity to generate a certain interval.

The researchers also created a computer model to help them further understand this phenomenon. They began with a model of hundreds of neurons connected together in random ways, and then trained it to perform the same interval-producing task they had used to train animals, offering no guidance on how the model should perform the task.

They found that these neural networks ended up using the same strategy that they observed in the animal brain data. A key discovery was that this strategy only works if some of the neurons have nonlinear activity — that is, the strength of their output doesn’t constantly increase as their input increases. Instead, as they receive more input, their output increases at a slower rate.

Jazayeri now hopes to explore further how the brain generates the neural patterns seen during varying time intervals, and also how our expectations influence our ability to produce different intervals.

The research was funded by the Rubicon Grant from the Netherlands Scientific Organization, the National Institutes of Health, the Sloan Foundation, the Klingenstein Foundation, the Simons Foundation, the Center for Sensorimotor Neural Engineering, and the McGovern Institute.

A sense of timing

by Charles Jennings |

The ability to measure time and to control the timing of actions is critical for almost every aspect of behavior. Yet the mechanisms by which our brains process time are still largely mysterious.

We experience time on many different scales—from milliseconds to years— but of particular interest is the middle range, the scale of seconds over which we perceive time directly, and over which many of our actions and thoughts unfold.

“We speak of a sense of time, yet unlike our other senses there is no sensory organ for time,” says McGovern Investigator Mehrdad Jazayeri. “It seems to come entirely from within. So if we understand time, we should be getting close to understanding mental processes.”

Singing in the brain

Emily Mackevicius comes to work in the early morning because that’s when her birds are most likely to sing. A graduate student in the lab of McGovern Investigator Michale Fee, she is studying zebra finches, songbirds that learn to sing by copying their fathers. Bird song involves a complex and precisely timed set of movements, and Mackevicius, who plays the cello in her spare time, likens it to musical performance. “With every phrase, you have to learn a sequence of finger movements and bowing movements, and put it all together with exact timing. The birds are doing something very similar with their vocal muscles.”

A typical zebra finch song lasts about one second, and consists of several syllables, produced at a rate similar to the syllables in human speech. Each song syllable involves a precisely timed sequence of muscle commands, and understanding how the bird’s brain generates this sequence is a central goal for Fee’s lab. Birds learn it naturally without any need for training, making it an ideal model for understanding the complex action sequences that represent the fundamental “building blocks” of behavior.

Some years ago Fee and colleagues made a surprising discovery that has shaped their thinking ever since. Within a part of the bird brain called HVC, they found neurons that fire a single short burst of pulses at exactly the same point on every repetition of the song. Each burst lasts about a hundredth of a second, and different neurons fire at different times within the song. With about 20,000 neurons in HVC, it was easy to imagine that there would be specific neurons active at every point in the song, meaning that each time point could be represented by the activity of a handful of individual neurons.

Proving this was not easy—“we had to wait about ten years for the technology to catch up,” says Fee—but they finally succeeded last year, when students Tatsuo Okubo and Galen Lynch analyzed recordings from hundreds of individual HVC neurons, and found that they do indeed fire in a fixed sequence, covering the entire song period.

“We think it’s like a row of falling dominoes,” says Fee. “The neurons are connected to each other so that when one fires it triggers the next one in the chain.” It’s an appealing model, because it’s easy to see how a chain of activity could control complex action sequences, simply by connecting individual time-stamp neurons to downstream motor neurons. With the correct connections, each movement is triggered at the right time in the sequence. Fee believes these motor connections are learned through trial and error—like babies babbling as they learn to speak—and a separate project in his lab aims to understand how this learning occurs.

But the domino metaphor also begs another question: who sets up the dominoes in the first place? Mackevicius and Okubo, along with summer student Hannah Payne, set out to answer this question, asking how HVC becomes wired to produce these precisely timed chain reactions.

Mackevicius, who studied math as an undergraduate before turning to neuroscience, developed computer simulations of the HVC neuronal network, and Okubo ran experiments to test the predictions, recording from young birds at different stages in the learning process. “We found that setting up a chain is surprisingly easy,” says Mackevicius. “If we start with a randomly connected network, and some realistic assumptions about the “plasticity rules” by which synapses change with repeated use, we found that these chains emerge spontaneously. All you need is to give them a push—like knocking over the first domino.”

Their results also suggested how a young bird learns to produce different syllables, as it progresses from repetitive babbling to a more adult-like song. “At first, there’s just one big burst of neural activity, but as the song becomes more complex, the activity gradually spreads out in time and splits into different sequences, each controlling a different syllable. It’s as if you started with lots of dominos all clumped together, and then gradually they become sorted into different rows.”

Does something similar happen in the human brain? “It seems very likely,” says Fee. “Many of our movements are precisely timed—think about speaking a sentence or performing a musical instrument or delivering a tennis serve. Even our thoughts often happen in sequences. Things happen faster in birds than mammals, but we suspect the underlying mechanisms will be very similar.”

Speed control

One floor above the Fee lab, Mehrdad Jazayeri is also studying how time controls actions, using humans and monkeys rather than birds. Like Fee, Jazayeri comes from an engineering background, and his goal is to understand, with an engineer’s level of detail, how we perceive time and use it flexibly to control our actions.

To begin to answer this question, Jazayeri trained monkeys to remember time intervals of a few seconds or less, and to reproduce them by pressing a button or making an eye movement at the correct time after a visual cue appears on a screen. He then recorded brain activity as the monkeys perform this task, to find out how the brain measures elapsed time. “There were two prominent ideas in the field,” he explains. “One idea was that there is an internal clock, and that the brain can somehow count the accumulating ticks. Another class of models had proposed that there are multiple oscillators that come in and out of phase at different times.”

When they examined the recordings, however, the results did not fit either model. Despite searching across multiple brain areas, Jazayeri and his colleagues found no sign of ticking or oscillations. Instead, their recordings revealed complex patterns of activity, distributed across populations of neurons; moreover, as the monkey produced longer or shorter intervals, these activity patterns were stretched or compressed in time, to fit the overall duration of each interval. In other words, says Jazayeri, the brain circuits were able to adjust the speed with which neural signals evolve over time. He compares it to a group of musicians performing a complex piece of music. “Each player has their own part, which they can play faster or slower depending on the overall tempo of the music.”

Ready-set-go

Jazayeri is also using time as a window onto a broader question—how our perceptions and decisions are shaped by past experience. “It’s one of the great questions in neuroscience, but it’s not easy to study. One of the great advantages of studying timing is that it’s easy to measure precisely, so we can frame our questions in precise mathematical ways.”

The starting point for this work was a deceptively simple task, which Jazayeri calls “Ready-Set-Go.” In this task, the subject is given the first two beats of a regular rhythm (“Ready, Set”) and must then generate the third beat (“Go”) at the correct time. To perform this task, the brain must measure the duration between Ready and Set and then immediately reproduce it.

Humans can do this fairly accurately, but not perfectly—their response times are imprecise, presumably because there is some “noise” in the neural signals that convey timing information within the brain. In the face of this uncertainty, the optimal strategy (known mathematically as Bayesian Inference) is to bias the time estimates based on prior expectations, and this is exactly what happened in Jazayeri’s experiments. If the intervals in previous trials were shorter, then people tend to under-estimate the next interval, whereas if the previous intervals were longer, they will over-estimate. In other words, people use their memory to improve their time estimates.

Monkeys can also learn this task and show similar biases, providing an opportunity to study how the brain establishes and stores these prior expectations, and how these expectations influence subsequent behavior. Again, Jazayeri and colleagues recorded from large numbers of neurons during the task. These patterns are complex and not easily described in words, but in mathematical terms, the activity forms a geometric structure known as a manifold. “Think of it as a curved surface, analogous to a cylinder,” he says. “In the past, people could not see it because they could only record from one or a few neurons at a time. We have to measure activity across large numbers of neurons simultaneously if we want to understand the workings of the system.”

Computing time

To interpret their data, Jazayeri and his team often turn to computer models based on artificial neural networks. “These models are a powerful tool in our work because we can fully reverse-engineer them and gain insight into the underlying mechanisms,” he explains. His lab has now succeeded in training a recurrent neural network that can perform the Ready-Set-Go task, and they have found that the model develops a manifold similar to the real brain data. This has led to the intriguing conjecture that memory of past experiences can be embedded in the structure of the manifold.

Jazayeri concludes: “We haven’t connected all the dots, but I suspect that many questions about brain and behavior will find their answers in the geometry and dynamics of neural activity.” Jazayeri’s long-term ambition is to develop predictive models of brain function. As an analogy, he says, think of a pendulum. “If we know its current state—its position and speed—we can predict with complete confidence what it will do next, and how it will respond to a perturbation. We don’t have anything like that for the brain—nobody has been able to do that, not even the simplest brain functions. But that’s where we’d eventually like to be.”

A clock within the brain?

It is not yet clear how the mechanisms studied by Fee and Jazayeri are related. “We talk together often, but we are still guessing how the pieces fit together,” says Fee. But one thing they both agree on is the lack of evidence for any central clock within the brain. “Most people have this intuitive feeling that time is a unitary thing, and that there must be some central clock inside our head, coordinating everything like the conductor of the orchestra or the clock inside your computer,” says Jazayeri. “Even many experts in the field believe this, but we don’t think it’s right.” Rather, his work and Fee’s both point to the existence of separate circuits for different time-related behaviors, such as singing. If there is no clock, how do the different systems work together to create our apparently seamless perception of time? “It’s still a big mystery,” says Jazayeri. “Questions like that are what make neuroscience so interesting.”

 

How the brain keeps time

Anne Trafton |

Keeping track of time is critical for many tasks, such as playing the piano, swinging a tennis racket, or holding a conversation. Neuroscientists at MIT and Columbia University have now figured out how neurons in one part of the brain measure time intervals and accurately reproduce them.

The researchers found the lateral intraparietal cortex (LIP), which plays a role in sensorimotor function, represents elapsed time, as animals measure and then reproduce a time interval. They also demonstrated how the firing patterns of population of neurons in the LIP could coordinate sensory and motor aspects of timing.

LIP is likely just one node in a circuit that measures time, says Mehrdad Jazayeri, the lead author of a paper describing the work in the Oct. 8 issue of Current Biology.

“I would not conclude that the parietal cortex is the timer,” says Jazayeri, an assistant professor of brain and cognitive sciences at MIT and a member of the McGovern Institute for Brain Research. “What we are doing is discovering computational principles that explain how neurons’ firing rates evolve with time, and how that relates to the animals’ behavior in single trials. We can explain mathematically what’s going on.”

The paper’s senior author is Michael Shadlen, a professor of neuroscience and member of the Mortimer B. Zuckerman Mind Brain Behavior Institute at Columbia University.

As time goes by

Jazayeri, who joined the MIT faculty in 2013, began studying timing in the brain several years ago while a postdoc at the University of Washington. He began by testing humans’ ability to measure and reproduce time using a task called “ready, set, go.” In this experiment, the subject measures the time between two flashes (“ready” and “set”) and then presses a button (“go”) at the appropriate time — that is, after the same amount of time that separated the “ready” and “set.”

From these studies, he discovered that people do not simply measure an interval and then reproduce it. Rather, after measuring an interval they combine that measurement, which is imprecise, with their prior knowledge of what the interval could have been. This prior knowledge, which builds up as they repeat the task many times, allows people to reproduce the interval more accurately.

“When people reproduce time, they don’t seem to use a timer,” Jazayeri says. “It’s an active act of probabilistic inference that goes on.”

To find out what happens in the brain during this process, Jazayeri recorded neuronal activity in the LIP of monkeys trained to perform the same task. In these recordings, he found distinctive patterns in the measurement phase (the interval between “ready” and “set”), and the production phase (the interval between “set” and “go”).

During the measurement phase, neuron activity increases, but not linearly. Instead, the slope of activity begins as a steep curve that gradually flattens out as time goes by, until the “set” signal is given. This is key because the slope at the end of the measurement interval predicts the slope of activity in the production phase.

When the interval is short, the slope during the second phase is steep. This allows the activity to increase quickly so that the animal can produce a short interval. When the interval is longer, the slope is gentler and it takes longer to reach the time of response.

“As time goes by during the measurement, the animal knows that the interval that it has to produce is longer and therefore requires a shallower slope,” Jazayeri says.

Using this data, the researchers could correctly predict, based on the slope at the end of the measurement phase, when the animal would produce the “go” signal.

“Previous research has shown that some neurons exhibit a ramping up of their firing rate that culminates with the onset of a timed motor response. This research is exciting because it provides the first hint as to what may control the slope of this ‘neural ramping,’ specifically that the slope of the ramp may be determined by the firing rate at the beginning of the timed interval,” says Dean Buonomano, a professor of behavioral neuroscience at the University of California at Los Angeles who was not involved in the research.

“A highly distributed problem”

All cognitive and motor functions rely on time to some extent. While LIP represents time during interval reproduction, Jazayeri believes that tracking time occurs throughout brain circuits that connect subcortical structures such as the thalamus, basal ganglia, and cerebellum to the cortex.

“Timing is going to be a highly distributed problem for the brain. There’s not going to be one place in the brain that does timing,” he says.

His lab is now pursuing several questions raised by this study. In one follow-up, the researchers are investigating how animals’ behavior and brain activity change based on their expectations for how long the first interval will last.

In another experiment, they are training animals to reproduce an interval that they get to measure twice. Preliminary results suggest that during the second interval, the animals refine the measurement they took during the first interval, allowing them to perform better than when they make just one measurement.

Mehrdad Jazayeri to join McGovern Institute faculty

by McGovern Institute |

We are pleased to announce the appointment of Mehrdad Jazayeri as an Investigator at the McGovern Institute for Brain Research. He will join the institute in January 2013, with a faculty appointment as assistant professor in MIT’s Department of Brain and Cognitive Sciences.

Complex behaviors rely on a combination of sensory evidence, prior experience and knowledge about potential costs and benefits. Jazayeri’s research is focused on the neural mechanisms that enable the brain to integrate these internal and external cues and to produce flexible goal-directed behavior.

In his dissertation work with J. Anthony Movshon at New York University, Jazayeri asked how the brain uses unreliable sensory signals to make probabilistic inferences. His work led to a simple computational scheme that explained how information in visual cortical maps is used for a variety of visual perceptual tasks. Later, as a Helen Hay Whitney postdoctoral fellow, he began to investigate the role of prior experience on perception. Working in the laboratory of Michael Shadlen at the University of Washington, he used a simple timing task to show that humans exploit their prior experience of temporal regularities to make better estimates of time intervals. Using a rigorous mathematical framework — Bayesian estimation — this work provided a detailed model for quantifying how measurements, prior expectations and internal goals influence timing behavior.

Jazayeri then turned to monkey electrophysiology to study how neurons process timing information and how they combine sensory cues with prior experience. For this work, he taught monkeys to reproduce time intervals, as if keeping the beat in music. The animals were provided with beats 1 and 2 and were rewarded for producing a third beat at the correct time. By recording from sensorimotor neurons in the parietal cortex during this task, Jazayeri showed that the pattern of activity is very different during the measurement and production phases of the task, even though the interval is the same.  Moreover, he found that the response dynamics of parietal neurons were shaped not only by the immediate time cues but also by the intervals monkeys had encountered in preceding trials.

Building on his previous work, Jazayeri will pursue two long-term research themes at MIT. One line of research will examine how brain circuits measure and produce time, an ability that is crucial for mental capacities such as learning causes and effects, “intuitive physics,” and sequencing thoughts and actions. The other line of research will exploit timing tasks to understand the neural basis of sensorimotor integration, a key component of cognitive functions such as deliberation and probabilistic reasoning.

Understanding complex behaviors such as flexible timing or sensorimotor integration requires methods for manipulating the activity of specific structures and circuits within the brain. Optogenetics, the ability to control brain activity using light, has emerged as a powerful tool for such studies. In a recent collaboration with Greg Horwitz at the Univeristy of Washington, Jazayeri reported the first successful application of optogenetics to evoke a behavioral response in primates. Motivated by this proof-of-principle experiment, Jazayeri plans to combine the traditional tools of psychophysics and electrophysiology with optogenetic manipulations to characterize the circuits that control timing and sensorimotor integration in the primate brain.

Originally from Iran, Jazayeri obtained his B.Sc in Electrical Engineering from Sharif University of Technology in Tehran. He received his PhD from New York University, where he studied with J. Anthony Movshon, winning the Dean’s award for the most outstanding dissertation in the university.  After graduating, he was awarded a Helen Hay Whitney fellowship to join the laboratory of Michael Shadlen at the University of Washington, where he has been since 2007.