We entirely agree with the central thrust of the article. But a broader view of what a “model” is, is needed.
In most of the examples discussed in the target article, a “model” is a generative system that synthesizes a specified output. For example, the target article discusses a system built by Lake et al. (Reference Lake, Salakhutdinov and Tenenbaum2015a) that learns to recognize handwritten characters from one or two examples, by modeling the sequence of strokes that produced them. The result is impressive, but the approach – identifying elements from a small class of items based on a reconstruction of how something might be generated – does not readily generalize in many other situations. Consider, for example, how one might recognize a cat, a cartoon of a cat, a painting of a cat, a marble sculpture of a cat, or a cloud that happens to look like a cat. The causal processes that generated each of these are very different; and yet a person familiar with cats will recognize any of these depictions, even if they know little of the causal processes underlying sculpture or the formation of clouds. Conversely, the differences between the causal processes that generate a cat and those that generate a dog are understood imperfectly, even by experts in developmental biology, and hardly at all by laypeople. Yet even children can readily distinguish dogs from cats. Likewise, where children learn to recognize letters significantly before they can write them at all well,Footnote
1
it seems doubtful that models of how an image is synthesized, play any necessary role in visual recognition even of letters, let alone of more complex entities. Lake et al.'s results are technically impressive, but may tell us little about object recognition in general.
The discussion of physical reasoning here, which draws on studies such as Battaglia et al. (Reference Battaglia, Hamrick and Tenenbaum2013), Gerstenberg et al. (Reference Gerstenberg, Goodman, Lagnado, Tenenbaum, Noelle, Dale, Warlaumont, Yoshimi, Matlock, Jennings and Maglio2015), and Sanborn et al. (Reference Sanborn, Mansingkha and Griffiths2013), may be similarly misleading. The target article argues that the cognitive processes used for human physical reasoning are “intuitive physics engines,” similar to the simulators used in scientific computation and computer games. But, as we have argued elsewhere (Davis & Marcus Reference Davis and Marcus2014; Reference Davis and Marcus2016), this model of physical reasoning is much too narrow, both for AI and for cognitive modeling.
First, simulation engines require both a precise predictive theory of the domain and a geometrically and physically precise description of the situation. Human reasoners, by contrast, can deal with information that is radically incomplete. For example, if you are carrying a number of small creatures in a closed steel box, you can predict that as long as the box remains completely closed, the creatures will remain inside. This prediction can be made without knowing anything about the creatures and the way they move, without knowing the initial positions or shapes of the box or the creatures, and without knowing the trajectory of the box.
Second, simulation engines predict how a system will develop by tracing its state in detail over a sequence of closely spaced instances. For example, Battaglia et al. (Reference Battaglia, Hamrick and Tenenbaum2013) use an existing physics engine to model how humans reason about an unstable tower of blocks collapsing to the floor. The physics engine generates a trace of the exact positions, velocities of every block, and the forces between them, at a sequence of instants a fraction of a second apart. There is no evidence that humans routinely generate comparably detailed traces or even that they are capable of doing so. Conversely, people are capable of predicting characteristics of an end state for problems where it is impossible to predict the intermediate states in detail, as the example of the creatures in the box illustrates.
Third, there is extensive evidence that in many cases where the actual physics is simple, humans make large, systematic errors. For example, a gyroscope or a balance beam constructed of solid parts is governed by the identical physics as the falling tower of blocks studied in Battaglia et al. (Reference Battaglia, Hamrick and Tenenbaum2013); the physical interactions and their analysis are much simpler for these than for the tower of blocks, and the physics engine that Battaglia et al. used in their studies will handle the case of a gyroscope or a balance beam without difficulty. But here, the model is “too good” relative to humans. Human subjects often make errors in predicting the behavior of a balance beam (Siegler Reference Siegler1976), and most people find the behavior of a gyroscope mystifying. Neither result follows from the model.
Intuitive psychology goes even further beyond what can be explained by sorts of generative models of action choice, discussed in the target article. One's knowledge of the state of another agent's mind and one's ability to predict their action are necessarily extremely limited; nonetheless, powerful psychological reasoning can be carried out. For example, if you see a person pick up a telephone and dial, it is a good guess that they he or she is planning to talk to someone. To do so, one does not need a full causal model of whom they want to talk to, what they will say, or what their goal is in calling. In this instance (and many others), there seems to be a mismatch between the currency of generative models and the sorts of inferences that humans can readily make.
So whereas we salute Lake et al.'s interest in drawing inferences from small amounts of data, and believe as they do that rich models are essential to complex reasoning, we find their view of causal models to be too parochial. Reasoning in humans, and in general artificial intelligence, requires bringing to bear knowledge across an extraordinarily wide range of subjects, levels of abstraction, and degrees of completeness. The exclusive focus on causal generative models is unduly narrow.
We entirely agree with the central thrust of the article. But a broader view of what a “model” is, is needed.
In most of the examples discussed in the target article, a “model” is a generative system that synthesizes a specified output. For example, the target article discusses a system built by Lake et al. (Reference Lake, Salakhutdinov and Tenenbaum2015a) that learns to recognize handwritten characters from one or two examples, by modeling the sequence of strokes that produced them. The result is impressive, but the approach – identifying elements from a small class of items based on a reconstruction of how something might be generated – does not readily generalize in many other situations. Consider, for example, how one might recognize a cat, a cartoon of a cat, a painting of a cat, a marble sculpture of a cat, or a cloud that happens to look like a cat. The causal processes that generated each of these are very different; and yet a person familiar with cats will recognize any of these depictions, even if they know little of the causal processes underlying sculpture or the formation of clouds. Conversely, the differences between the causal processes that generate a cat and those that generate a dog are understood imperfectly, even by experts in developmental biology, and hardly at all by laypeople. Yet even children can readily distinguish dogs from cats. Likewise, where children learn to recognize letters significantly before they can write them at all well,Footnote 1 it seems doubtful that models of how an image is synthesized, play any necessary role in visual recognition even of letters, let alone of more complex entities. Lake et al.'s results are technically impressive, but may tell us little about object recognition in general.
The discussion of physical reasoning here, which draws on studies such as Battaglia et al. (Reference Battaglia, Hamrick and Tenenbaum2013), Gerstenberg et al. (Reference Gerstenberg, Goodman, Lagnado, Tenenbaum, Noelle, Dale, Warlaumont, Yoshimi, Matlock, Jennings and Maglio2015), and Sanborn et al. (Reference Sanborn, Mansingkha and Griffiths2013), may be similarly misleading. The target article argues that the cognitive processes used for human physical reasoning are “intuitive physics engines,” similar to the simulators used in scientific computation and computer games. But, as we have argued elsewhere (Davis & Marcus Reference Davis and Marcus2014; Reference Davis and Marcus2016), this model of physical reasoning is much too narrow, both for AI and for cognitive modeling.
First, simulation engines require both a precise predictive theory of the domain and a geometrically and physically precise description of the situation. Human reasoners, by contrast, can deal with information that is radically incomplete. For example, if you are carrying a number of small creatures in a closed steel box, you can predict that as long as the box remains completely closed, the creatures will remain inside. This prediction can be made without knowing anything about the creatures and the way they move, without knowing the initial positions or shapes of the box or the creatures, and without knowing the trajectory of the box.
Second, simulation engines predict how a system will develop by tracing its state in detail over a sequence of closely spaced instances. For example, Battaglia et al. (Reference Battaglia, Hamrick and Tenenbaum2013) use an existing physics engine to model how humans reason about an unstable tower of blocks collapsing to the floor. The physics engine generates a trace of the exact positions, velocities of every block, and the forces between them, at a sequence of instants a fraction of a second apart. There is no evidence that humans routinely generate comparably detailed traces or even that they are capable of doing so. Conversely, people are capable of predicting characteristics of an end state for problems where it is impossible to predict the intermediate states in detail, as the example of the creatures in the box illustrates.
Third, there is extensive evidence that in many cases where the actual physics is simple, humans make large, systematic errors. For example, a gyroscope or a balance beam constructed of solid parts is governed by the identical physics as the falling tower of blocks studied in Battaglia et al. (Reference Battaglia, Hamrick and Tenenbaum2013); the physical interactions and their analysis are much simpler for these than for the tower of blocks, and the physics engine that Battaglia et al. used in their studies will handle the case of a gyroscope or a balance beam without difficulty. But here, the model is “too good” relative to humans. Human subjects often make errors in predicting the behavior of a balance beam (Siegler Reference Siegler1976), and most people find the behavior of a gyroscope mystifying. Neither result follows from the model.
Intuitive psychology goes even further beyond what can be explained by sorts of generative models of action choice, discussed in the target article. One's knowledge of the state of another agent's mind and one's ability to predict their action are necessarily extremely limited; nonetheless, powerful psychological reasoning can be carried out. For example, if you see a person pick up a telephone and dial, it is a good guess that they he or she is planning to talk to someone. To do so, one does not need a full causal model of whom they want to talk to, what they will say, or what their goal is in calling. In this instance (and many others), there seems to be a mismatch between the currency of generative models and the sorts of inferences that humans can readily make.
So whereas we salute Lake et al.'s interest in drawing inferences from small amounts of data, and believe as they do that rich models are essential to complex reasoning, we find their view of causal models to be too parochial. Reasoning in humans, and in general artificial intelligence, requires bringing to bear knowledge across an extraordinarily wide range of subjects, levels of abstraction, and degrees of completeness. The exclusive focus on causal generative models is unduly narrow.