Show Less
Open access

In the Beginning was the Image: The Omnipresence of Pictures

Time, Truth, Tradition


András Benedek and Ágnes Veszelszki

The authors outline the topic of visuality in the 21st century in a trans- and interdisciplinary theoretical frame from philosophy through communication theory, rhetoric and linguistics to pedagogy. As some scholars of visual communication state, there is a significant link between the downgrading of visual sense making and a dominantly linguistic view of cognition. According to the concept of linguistic turn, everything has its meaning because we attribute meaning to it through language. Our entire world is set in language, and language is the model of human activities. This volume questions the approach in the imagery debate.

Show Summary Details
Open access

Space and Action to Reason: from Gesture to Mathematics (Valeria Giardino)

← 40 | 41 →

Valeria Giardino

Space and Action to Reason: from Gesture to Mathematics

1.  The Target: Tools for Thought

In our cultural evolution, humans have created several instruments, not only with the intention of surpassing their physical limits – think of a hammer or a pair of glasses – but also with the aim of enhancing their memory and inferential powers; consider for example a map or the abacus. In the present paper I will focus on tools for thought. How do they work? Or, to clarify, how do we work with them?

In her article in the fourth volume of the Series Visual Learning, Krämer (2014) has pointed out that the human sciences have generally focused on a crucial dichotomy between word and image. However, tools for thought do not seem to fit this binary ordering: tables, graphs, diagrams or maps, arise as a “conjunction of language and image,” which is very powerful since it makes “saying” and “showing” work together, thus reaching beyond the word/image binary opposition. Such a conjunction establishes an “operative iconicity”: cognitive artefacts are not only intended as representations (a map represents a city and a Venn diagram represents certain logical relations) but they are also inviting their user to rely on them as instruments to generate and further explore the represented objects of knowledge (we navigate the city through a map and we extract new logical relations by reconfiguring a Venn diagram).

Writing as well can be considered a cognitive artefact.1 Goody (2000) took the experience of the protagonist of Defoe’s Robinson Crusoe as paradigmatic: it is only where he finds pen, ink and paper that he finally begins “keeping things very exact,” most of all by telling the time. In Goody’s view, writing has been a crucial improvement in our cultural evolution, since it has influenced not only the content of our exchanges but their very structure. The drawing of the line has been a graphic accomplishment that has changed forever the way we formulate our thoughts (think for example of the line constituted by the carvings in a stone to track days passing by). Writing and the drawing of conceptual lines are ← 41 | 42 → phenomena that cannot be considered from within the word/image dichotomy. Despite this, operative iconicity brings about a form of “cognitive technology” that becomes available for sharing and structuring the content of our thoughts. As Goody suggests, we could think for example of the mathematical table – certainly a product of writing but more interestingly of the drawing of the graphical line – as a tool for thought whose functioning can be taught to and learned also by people who can neither read nor write. The table here is “a technology for the intellect”; in Krämer’s words: it belongs to the class of the “diagrammatic,” overcoming the word/image dichotomy.

In the present paper, I will assume that “operative iconicity” is indeed the characteristic feature of numerous tools for thought and my aim will be to explain the conditions for it. In Section 2, I will introduce my hypothesis about the existence of a human capacity labelled diagramming. In Section 3, I will discuss diagramming in relation to the philosophy of mind. In Section 4, I will present some pieces of evidence coming from experimental psychology and philosophy of mathematics supporting the diagramming hypothesis. Finally, in Section 5 I will briefly discuss some open questions.

2.  The Hypothesis: Diagramming

In an article in the fourth volume of this series, I introduced my hypothesis about the existence of a capacity that I have labelled diagramming (Giardino 2014). Thanks to this capacity, humans are able to recruit a variety of cognitive systems – spatial perception and action systems – that are already available in other contexts, with the specific aim of reducing cognitive loads for memory and assisting problem solving. The human species is endowed with natural capacities for treating spatial information and planning/regulating action in the external world, which have an undisputable survival value. In my view, by connecting the systems responsible for spatial perception and action – and possibly other systems – humans have invented a class of (cognitive) artefacts capable of assisting them in solving new problems originated in the course of their cultural evolution – the nature of which would have made it impossible to address them using one of those systems alone. In particular, according to my hypothesis: (i) the production and the analysis of spatial cognitive artefacts recruit action systems; (ii) in some cases, recruiting action systems makes problem solving easier (problems are solved more efficiently, more quickly, and with less cognitive efforts); (iii) spatial cognitive artefacts trigger this recruiting and therefore they act as multi-recruiting systems.

In line with Krämer, I argue that to give an account of the functioning of cognitive tools it is necessary to go beyond the dichotomy word/image. In fact, spatial ← 42 | 43 → cognitive artefacts are not simply visual tools but dynamic devices, allowing for new inferences by being acted upon; tools for thought are the medium where space perception and orientation, action planning and regulation and other cognitive systems (e.g. visual and conceptual) operate in coordination in view of a cognitive task. For this reason, they can gain us a unique insight on the functioning of the human mind and on its motives and capacities for cultural innovation.

Imagine performing an action on a geometric figure, such as translating or rotating it. This will bring together various cognitive systems by triggering each of them: in that same action, the conceptual system will recognize the conformity to some invariance; the visuo-spatial system a transformation in time; and the action system a movement or a movement plan (Giardino 2014). The connection among these different perspectives will allow for the performance of the appropriate inference. Thanks to diagramming, the physical properties of a particular representation are interpreted as referring to other elements – abstract as well – which are not directly present in the space of the representation: by manipulating physical properties, it will be in some cases possible to learn something new about the objects or the events which they refer to. The products of the diagramming capacity are “iconic,” that is, conceived to structure the space of the problem. By using the term “iconic,” the reference is to the notion of “icon” as introduced by Peirce. As Peirce suggests, a “diagram-icon” is a tool that allows making inferences because it has been constructed with an intention and is dynamic: the diagram-icon leads its interpretant to a state of activity that, mingled to curiosity, brings one to experimentation (Peirce [c. 1906]). It is an “operative iconicity,” to use Krämer’s term again.

3.  The Background: Moderate Embodiment

In the recent philosophical literature, various hypotheses about the nature of our mind have been put forward that are possibly in line with the diagramming hypothesis. For example, Varela et al. (1991) have proposed that the mind is embodied, and Clark and Chalmers (1998) that it is extended. Despite being very inspiring, the metaphors of the mind as embodied or extended do not seem unfortunately to be enough when it comes to clarifying, in practice, how humans were able to create and are able to rely on ‘scaffolding’ structures in order to enhance their reasoning and inferential capacities. I do not take here any particular metaphysical stance about the location of the mind in the environment; more modestly, I claim that cognitive processes do not happen exclusively in the brain but extend themselves beyond the skin and skull of an individual, and therefore cognition happens to be distributed between internal and external structures, for ← 43 | 44 → example, in space (see for reference Hutchins 2001). It still remains to evaluate the possible role of the body: how much the experience of having a particular body would matter for the performance of a variety of cognitive tasks?

On this topic, Goldman (2012) has interestingly proposed to assume a moderate approach to embodied cognitive science. According to Goldman, empirical studies show that human cognition can in fact be considered for the most part as embodied because information obtained from proprioception and kinesthetics (which refer in the first place to the perception of the position and of the movements of the different parts of the body), happens to be often reused for other, more abstract tasks. The empirical part of Goldman’s claim is that there exist so-called bodily representational codes, that is, a subset of mental codes that are primarily or fundamentally applied in forming interoceptive or directive representations of one’s own bodily states and activities (see for reference Goldman–Vignemont 2009). The philosophical part of Goldman’s claim is that the brain reuses or redeploys cognitive processes having different original use to solve new tasks in new contexts; when it comes to bodily representational codes, these appear to be extremely pervasive: selected cognitive tasks might be executed via embodied processes, and there is no need to ascend to more global claims.

Goldman’s proposal leads to a “moderate” conception of embodiment-oriented cognitive science precisely because it specifies the role of the body in cognition, first by defining what bodily representational codes are and then by explaining how they happen to have an influence on some cognitive processes. This framework is crucial to consider diagramming as allowing us to perform actions on material spatial artefacts, leading to cognitive advantages in some particular tasks. Such a view runs counter to standard theories in cognitive science, which have claimed that core knowledge representations in cognition are amodal data structures that get processed independently of the brain’s modal systems for perception, action and introspection.

4.  Some Evidence from Mathematics

4.1  Gestures in Explanation

What if mathematics in particular is (moderately) embodied? As a first piece of evidence in favour of the existence of the diagramming capacity, consider gesture in mathematics. According to standard approaches, mathematics is an abstract science, dealing with abstract objects described by signs not sharing any properties with the ordinary objects we interact with every day. However, some recent studies have questioned the existence of a sharp distinction between abstract knowledge ← 44 | 45 → on the one hand and the concrete world on the other. According to Lakoff and Núñez (2000), abstract mathematical concepts are rooted in embodied activities, such as for example our ways of thinking about the world, and how we describe it, that is, our perception. In line with previous studies on language (see for reference Lakoff–Johnson 1980), the authors propose that abstract scientific concepts, as well as ordinary ones, can be reformulated in terms of metaphors, which are not mere linguistic phenomena, but crucial elements of thought. Indeed, the typical mathematical jargon contains many terms that allude to our relationship with the real world: natural numbers “grow” indefinitely, points “lay” on a line, functions “move” to zero. The basis of these metaphors is bodily experience: mathematics is embodied because we understand and explain it by making appeal to embodied cognitive mechanisms, of which conceptual metaphor is the main one.

Recently, Sinclair and Gol Tabaghi (2010) have interviewed six mathematicians and asked them to explain the meaning of the mathematical concept of “eigenvector”. The interviews were filmed, with the aim of evaluating the mathematicians’ embodied reasoning. The videos show that the mathematicians make use of a variety of representations – speech, gestures, diagrams, and so on – moving from one to the other without difficulties, thus blurring the alleged boarders between the mathematical and the physical world. They all well know the formal ‘manual’ definition of eigenvector, single, atemporal and static; nonetheless, without exception, they offer a description of eigenvectors that alludes to a very different interpretation, including also temporal and kinaesthetic elements, as shown both in the terms and in the gestures they use. Metaphors are common: some of the mathematicians focus on the transformations of the vectors, by saying and showing in their gestures how they “shrink” or “turn”; others describe the vectors’ personality, by claiming that they “go in the same direction,” they “align,” and so on. In other words, none of the mathematicians speaks of eigenvectors only in terms of algebraic equalities. Moreover, some metaphors are perceptual (for example, one mathematician thinks of the quadratic function as a “goblet”) but most of them have a movement component. Furthermore, the high variability in their gestures seems to depend on their respective education and competence, i.e. on their relevant background knowledge. Gestures, compared to language, are in some sense the ‘degree zero’ of diagramming: they give more possibilities than simple speech to express continuity, time and movement, thus confirming the intuition of the French mathematician Châtelet (1993), who saw in a mathematical diagram the “crystallization” of a gesture. ← 45 | 46 →

4.2  Formulas in Algebra

A second piece of evidence in favour of the existence of diagramming comes from recent experimental work on mathematical notations. Landy and Goldstone (2007) have considered how physical layout affects the segmentation of simple equations. A difficult and yet routine part of mathematical reasoning is to segment a notational form, that is, to parse it into its formal components. For example, in a formula, in the absence of parentheses, multiplication comes before addition, and therefore the equation must be parsed accordingly. Following the standard approaches to mathematical reasoning, the cognitive parser executes segmentation by applying formal rules to individual notational symbols; the assumption is that abstract symbol sequences are trivially extracted from physical notations. To run counter to this assumption, the authors added several visual cues such as spacing, lines or circles to the formulas of an equation and then showed them to their subjects. The hypothesis was that such cues would trigger the application of perceptual grouping mechanisms and as a consequence the capacity for symbolic reasoning. The results showed indeed that the subjects’ judgments about the validity of the equations were more likely to be correct if visual groupings were in line with valid operator precedence (multiplication must be executed before addiction in the absence of parentheses). This would give evidence to the authors’ hypothesis that people use typically available non-formal information to make grouping judgments, and only subsequently integrate this information with formal rules, which is in line with diagramming.

4.3  Diagrams in Topology

A third piece of evidence in favour of diagramming comes from my research about visual tools in topology. Several philosophers have discussed the use of diagrams in mathematics (Giardino [forthcoming]): mathematicians often rely on the space of the representations they use in some cases by performing some actions on them. Space is crucial for reasoning: as it is well known, in the 1980s Johnson-Liard (1983) proposed a theory of human reasoning that was based on mental models, that is, human reasoning would be a mental simulation process in which models of the premises are constructed, inspected, and validated. However, according to the diagramming hypothesis, spatial cognitive artefacts are multi-recruiting systems, and therefore also the actions performed are crucial for reasoning. Kirsh and Maglio (1994) have famously proposed to distinguish between pragmatic and epistemic actions: actions of the first kind aim to bring the agent closer to his or her physical goal, while actions of the second kind “use [the] world to improve cognition,” i.e., they are “physical actions that make ← 46 | 47 → mental computation easier” (Kirsh–Maglio 1994: 513). An epistemic action is performed outside on the physical objects that are available, and it is precisely the performance of this action that enhances our inferential capacities. The concept of epistemic action can be extended to the use of signs in mathematics. As De Cruz and De Smedt have recently proposed, mathematical symbols “enable us to perform mathematical operations that we would not be able to do in the mind alone, they are epistemic actions” (De Cruz–De Smedt 2013: 4). Of course, such a view is not epistemologically innocuous, since it assumes that mathematical signs are intimately linked to the concepts they represent and vice versa.

Recently, together with Silvia De Toffoli, we have considered the case of a practice of mathematics where concrete manipulations and transformations are performed on the space of the representations: the practice of proof in topology, in particular in knot theory and low dimensional topology (De Toffoli–Giardino 2014; 2015).2 In our interpretation, knot diagrams are dynamic tools: in perceiving a diagram; one has to see the possible moves that can be applied to them. For this reason, experts have developed a specific form of enhanced manipulative imagination, which allows them to draw inferences from knot diagrams by imagining, and in some cases actually performing, epistemic actions on them. We argued that the meaning of a knot diagram is not pre-defined before the definition of its context of use: it is only when the mathematicians fix the mathematical space in which the particular diagram is embedded and as a consequence choose which manipulations are allowed that the knot diagram becomes meaningful. This indeterminacy of meaning is precisely the feature that makes knot diagrams a space for experimentation, where different actions and consequently different possible interpretations can be tested. For this reason, knot diagrams are a good notation, because they represent mathematical concepts but at the same time have inferential power: a notation is “good” when it not only facilitates calculation but also prompts new ideas and induces new developments. Also the actual practice of proving in low-dimensional topology involves a kind of reasoning that cannot be reduced to formal statements without a loss of intuition: the representations that are commonly used in low-dimensional topology are heterogeneous, i.e. neither entirely propositional nor entirely visual (the word/image dichotomy does not apply). ← 47 | 48 →

Manipulative imagination is shared by experts: it is the kind of reasoning that one has to master to become a practitioner. Moreover, the manipulations allowed on the representations as well as the representations themselves are epistemologically relevant, since they are integral parts both of the reasoning and of the justification provided: the representations give a material form to the transformations (and in this sense they embody them), thus allowing experts performing epistemic actions (and in particular drawing inferences) on them. These actions are controlled by the shared practice: the set of legitimate transformations is limited and determined by the context. Differently from gestures, diagrams are not idiosyncratic but part of a solid and stable practice.

5.  Open Problems

To sum up, according to the diagramming hypothesis, spatial cognitive artefacts act as multi-recruiting systems. Despite not being material, gestures show that spatial and motor elements might help comprehension. Moreover, it has been shown that the way the notation is perceived influences comprehension and that experts in topology are able to envision (physical) transformations on the diagrams having conceptual consequences.

If diagramming really exists, then two questions can be formulated in relation to the general topic of this collection. The first question is for people working in education: would it be better to replace the reference to visual learning with that of (moderately) embodied learning? The second question is for the developers: what happens (or maybe what has already happened) when diagramming meets the new technologies? For example, should touch screen technologies be designed in accordance to a more serious reflection about the role of space and action in reasoning? This is matter (I hope) of further research.


Châtelet, Gilles (1993): Les Enjeux du mobile: Mathématique, physique, philosophie. Paris: Collection Des Travaux, Edition du Seuil.

Clark, Andy – Chalmers, David John (1998): The Extended Mind. Analysis 58/1: 10–23.

De Cruz, Helen – De Smedt, Johan (2013): Mathematical symbols as epistemic actions. Synthese 190: 3–19.

De Toffoli, Silvia – Giardino, Valeria (2014): Forms and Roles of Diagrams in Knot Theory. Erkenntnis 79/4: 829–842. ← 48 | 49 →

De Toffoli, Silvia – Giardino, Valeria (2015): An Inquiry into the Practice of Proving in Low-Dimensional Topology. Boston Studies in the Philosophy and History of Science 308: 315–336.

De Toffoli, Silvia – Giardino, Valeria (2016): Envisioning Transformations. The Practice of Topology. In: Larvor, B. (ed.): Mathematical Cultures. Basel: Birkhäuser Science, Springer. 25–50.

Giardino, Valeria (2014): Diagramming: Connecting Cognitive Systems to Improve Reasoning. In: Benedek, András – Nyíri, Kristóf (eds.): Emotion, Expression, Explanation. Visual Learning, vol. 4. Frankfurt/M: Peter Lang Verlag. 23–34.

Giardino, Valeria (forthcoming 2017): Diagrammatic Reasoning in Mathematics. In: Magnani, L. – Bertolotti, T. (eds.): Spinger Handbook of Model-Based Science. Basel: Springer.

Goldman, Alvin (2012): A Moderate Approach to Embodied Cognitive Science. The Review of Philosophy and Psychology 3/1: 71–88.

Goldman, Alvin – Vignemont, Frederique de (2009): Is Social Cognition Embodied? Trends in Cognitive Sciences 13/4: 154–159.

Goody, Jack (2000): The Power of the Written Tradition. Washington: Smithsonian Institution Press.

Hutchins, Edwin (2001): Distributed Cognition. In: Smelser, N. J. – Baltes, P. B. (eds.): The International Encyclopedia of the Social and Behavioral Sciences. 2068–2072.

Johnson-Liard, Philip N. (1983): Mental Models. Cambridge: Cambridge University Press.

Kirsh, David – Maglio, Paul (1994): On Distinguishing Epistemic from Pragmatic Action. Cognitive Science 18: 513–549.

Krämer, Sybille (2014): Trace, Writing, Diagram: Reflections on Spatiality, Intuition, Graphical Practices and Thinking. In: Benedek, András – Nyíri, Kristóf (eds.): Emotion, Expression, Explanation. Visual Learning, vol. 4. Frankfurt/M: Peter Lang Verlag. 3–22.

Lakoff, George P. – Johnson, Mark (1980): Metaphors We Live By. Chicago: University of Chicago Press.

Lakoff, George P. – Nuñez, Rafael E. (2000): Where Mathematics Come From: How the Embodied Mind Brings Mathematics into Being. New York: Basic Books.

Landy, David – Goldstone Robert L. (2007): How Abstract is Symbolic Thought? Journal of Experimental Psychology: Learning, Memory, and Cognition 33/4: 720–733. ← 49 | 50 →

Peirce, Charles Sanders (1976 [c. 1906]): PAP (Prolegomena to an Apology for Pragmatism) (MS 293). In: Peirce, C. S.: The New Elements of Mathematics by Charles S. Peirce. V. 4. Edited by C. Eisele. The Hague, Netherlands: Mouton Publishers. 319–320.

Sinclair, Nathalie – Gol Tabaghi, Shiva (2010): Drawing Space: Mathematicians’ Kinetic Conceptions of Eigenvectors. Educational Studies in Mathematics 74/3: 223–240.

Varela, Francisco J. – Thompson, Evan – Rosch, Eleanor (1991): The Embodied Mind: Cognitive Science and Human Experience. Cambridge, MA: MIT Press.

1 In the following, I will use cognitive artifact as synonym for tool for thought. However, it is true that there might be cases of tools for thought that are not strictly speaking artifacts, for example gestures.

2 More recently and along the same line of thought, we have analysed the proof of “Alexander’s Theorem,” which allows connecting knots to topological braids (De Toffoli–Giardino 2016).