Foundational skills and dispositions for learning
An experience with Information Problem Solving on the Web”
Active participation in the information society requires the ability to find some order in the chaotic nature of the Web and not to get lost within the endemic presence of inaccurate, misleading, biased and false information. This article presents an approach to Information Problem Solving (IPS) – that is, finding, understanding and assessing information on the Web – and discusses a study carried out in an Italian secondary school, using an experimental and a control group. The study aimed at exploring how to best foster IPS skills, and observing whether and how IPS activities could promote the development of more general learning dispositions and competences. After a period of training with IPS activities, the experimental group showed different dispositions towards learning from a text and engaging with open-ended questions. Despite serious limitations in the depth of analysis, most students were able to reach acceptable solutions; at the same time, they felt empowered and developed an embryonic critical attitude on which it might be possible to build further.
Information Problem Solving, digital literacy, critical literacy, secondary school
Quote as: Caviglia, Francesco & Delfino, Manuela. (2016). Foundational skills and dispositions for learning: An experience with Information Problem Solving on the Web. Technology, Pedagogy and Education, 25(4), 487–512. https://doi.org/10.1080/1475939X.2015.1080756
Abstract
Active participation in the information society requires the ability to find some order in the chaotic nature of the Web and not to get lost within the endemic presence of inaccurate, misleading, biased and false information. This article presents an approach to Information Problem Solving (IPS) – that is, finding, understanding and assessing information on the Web – and discusses a study carried out in an Italian secondary school, using an experimental and a control group. The study aimed at exploring how to best foster IPS skills, and observing whether and how IPS activities could promote the development of more general learning dispositions and competences. After a period of training with IPS activities, the experimental group showed different dispositions towards learning from a text and engaging with open-ended questions. Despite serious limitations in the depth of analysis, most students were able to reach acceptable solutions; at the same time, they felt empowered and developed an embryonic critical attitude on which it might be possible to build further.
[p. 487]
Introduction
Being literate today implies – in addition to traditional print-based reading skills – being able to use computers for interacting with vast amounts of information accessed via search engines. In particular, being able to find, understand and assess information on the Web is crucial for active participation in the information society (Organisation for Economic Co-operation and Development [OECD], 2010a, 2012; Warschauer, 1999, 2002).
Throughout this article, we will refer to the range of processes involving finding, understanding and assessing information on the Web as ‘Information Problem Solving on the Web’ (henceforth, IPS). Solving information problems on the Web is a complex activity that requires parallel activation of a network of non-trivial competences, as described in Figure 1.
This article is based on a six-month study with IPS including Italian K–9 students (ages 14–15) which took place during the 2007/2008 academic year. The study was inspired by two research questions: the first was how to best foster IPS skills in teenage learners; the second was whether and how IPS activities could [p. 488] foster the development of more general learning competences, as suggested by a body of pioneer studies.
Classroom activities described in this study were carried out in a computer lab, as part of an experimental curriculum in ‘learning how to learn’. This means that, at the time the study took place, IPS required ad-hoc equipment and facilities that were separated from the traditional classroom. Seven years later, also thanks to the ubiquity and variety of tools for searching the Web in formal and informal learning situations, IPS is recognised as a core element of literacy (e.g. in the curricular national recommendations by the Italian Ministry of Education [MIUR], 2013). Meanwhile, skills and dispositions for lifelong learning have become a key element in exploring the connection between technology and future skills for learning and work (Bellanca & Brandt, 2010; Cheeseman, Sumsion, & Press, 2014; Collins & Halverson, 2009; Davies, Fidler, & Gorbis, 2011; Dee Fink, 2013; Selwyn, 2011; Thomas & Brown, 2011).
Therefore, better understanding of how a habit of solving well-designed information problems can reverberate on the students’ disposition towards learning is possibly more relevant today than at the time of the original study. This article thus presents results that are directly supported by data and observations from the original field study, filtered through more recent readings and supplemented by updates about new developments in the practice and technology of web use.
[p. 489]
Background assumptions
The impact of the Internet on learning practices implies moving from a ‘paucity economy’ to an ‘abundance economy’ of information (J. S. Brown, 2000; Thomas & Brown, 2011). In the first case, when access to information was more limited, the focus of learning was mastering existing information, and learning typically meant appropriating established knowledge through canonical texts. Being able to ‘read in depth’ was therefore the most valuable asset in the age of printing literacy (Simone, 2002, 2012).
Reading in depth is still a key to understanding and learning – possibly with a stronger focus on connecting information from different sources and media – but today’s information abundance regime has made other dimensions of literacy equally important requirements for lifelong learning: (a) retrieving, locating and evaluating information (Leu, Kinzer, Coiro, & Cammack, 2004; Ott & Meurers, 2010); (b) creating and sharing new knowledge, understanding and developing new artefacts (Bereiter & Scardamalia, 2005; Dennett, 2013; Jenkins, Purushotma, Weigel, Clinton, & Robison, 2009; Paavola, Engeström, & Hakkarainen, 2012; Scardamalia & Bereiter, 2006).
These two dimensions are deeply intertwined in most real-life learning settings. However, we decided to focus primarily on locating and evaluating information because of its foundational role; furthermore, there was – and still is – compelling evidence that a large number of young and adult computer users are not very competent at finding and evaluating information on the Internet (Caviglia & Ferraris, 2006; Hargittai, 2010; Hargittai, Fullerton, Menchen-Trevino, & Thomas, 2010; Kuiper, Volman, & Terwel, 2005; Lumley & Mendelovits, 2012; Walraven, Brand-Gruwel, & Boshuizen, 2008, 2009).
IPS on the Web
Initially, we took inspiration for exploring the use of the Web as a learning resource from studies that suggested how web-mediated problem-solving activities have the potential for fostering high-order competences such as asking questions (McKenzie, 1997), doubting (Burbules, 2000), using abductive reasoning (Cunningham, Arici, Schreiber, & Lee, 2002) and connecting as opposed to collecting (Moss, 2002). As for bringing these suggestions to the literacy classroom, we were especially intrigued by the idea that the Web intrinsically supported inquiry-based learning (Bruce & Bishop, 2002; Edelson, Gordin, & Pea, 1999). In this approach, the learner, instead of being assigned by the teacher simple questions or problems bounded to a unique solution, is confronted with questions that require looking for new information in order to find plausible answers, thereby reproducing a typical real-life setting. Inquiry-based learning activities have flourished especially in the area of natural sciences, where observation and measures provide a key source of information, but also in domains in which the learners collect information from the Web (e.g. Bruce, 2001; Childs, Sorensen, & Twidle, 2011; Inquiry Page, 2010, for examples).
In the two years that preceded the project described in this article, our research group explored several forms of web-based learning activities by trying them out personally, by assigning them to fellow teachers, researchers and our own students. We soon observed that when the Web was merely a tool for delivering contents (e.g. when the teacher provided a list of websites as learning material), the computer as [p. 490] medium was inferior to reading on paper (more distractions, sense of ‘getting lost’ in navigation, difficulties in marking the text and taking notes; see Hillesund, 2010). This was especially evident in activities focusing on learning subject-matter related contents. We experienced much richer learning experiences when we asked our learners to tackle real-life, ill-defined problems that were often thin in content but rich in the perspective of processes involved. The question, for instance, ‘Why are the colours of the Genoa football team red and blue?’ was not relevant in regard to any subject area, but sparked some interesting discussion and web searches among groups of students and teachers (until the answer eventually became integrated in Wikipedia, making the problem too easy to solve and therefore no longer useful as an IPS activity; Caviglia & Ferraris, 2006, 2008).
The crucial issue in developing IPS activities was – and still is – to design good problems, that is, problems that require some reflection before deciding what to search and how to evaluate the results. In these ideal cases, the Web provides a learning setting that supports the student in exploring and reaching a better understanding of the problem, without trivialising the task by delivering an easy-to-find solution.
Evaluating information in technology-rich environments
Based on a representative sample of 15-year-old students from OECD countries taking part in the OECD PISA (Programme for International Student Assessment) 2009 survey on digital literacy, Lumley and Mendelovits (2012, p. 9) underlined how ‘[most of them] do not know how to begin evaluating material they encounter on the internet’. These results suggest that the ability to evaluate information critically is not adequately integrated into school curricula, with students typically expected to understand and reflect on ‘good’ texts rather than question their trustworthiness. Indeed, being able to evaluate information has always been a distinctive trait of advanced literacy, but recognising trustworthiness has now become a foundational competence for using the Web (Leu, Kinzer, Coiro, & Cammack, 2004; OECD, 2010a).
The variable quality of information has often been regarded a problem for using the Web in educational settings. This attitude is evident, for example, in one popular approach to using the Web in education, WebQuests (Dodge, 1997): in its standard form, a WebQuest provides a list of pre-selected websites that the student is expected to visit to collect information useful for completing a task. Dodge explicitly writes: ‘Because pointers to resources are included, the learner is not left to wander through webspace completely adrift.’ In other words, the Web in its entirety is perceived as a potential waste of time or diversion.
This article suggests, instead, that it is precisely the chaotic nature of the Web fraught with inaccurate, misleading, biased and false information, that provides an excellent setting for practising how to assess information. The Web itself makes it easy to find alternative sources of information, while the presence of contrasting claims exemplifies the need for the reader to evaluate their trustworthiness.
An urgent caveat is necessary at this point. Developing a critical attitude towards sources of information is a complex process that typically requires domain-specific knowledge, accompanied by advanced competences in discourse analysis and scientific reasoning (Caviglia, 2002). The approach proposed in this article should be regarded as a very first step in this direction, with focus on checking facts rather [p. 491] than exposing rhetoric and ideology, as in the tradition of critical discourse analysis (Farirclough & Wodak, 1997), or faulty thinking patterns, as in the critical thinking approach (Halpern, 1998; see a comparison of the two approaches in Burbules & Berk, 1999). At the same time, as we will argue below in the Discussion, we believe that relatively simple problems related to the trustworthiness of a source provide a sensible point of departure for initiating and practising a critical attitude.
Bringing IPS to the Italian school context
In the Italian school tradition, students have typically been required to read and repeat textbook contents or to apply procedures that have been explained by the teacher. In other words, solving new problems in situations where no solution method is obvious to the problem solver – as required in the generally accepted definition of problem solving (Mayer & Wittrock, 2006, p. 287; OECD, 2010b, p. 11) – was relatively unusual. It did not come as a surprise, therefore, when OECD PISA 2003 results revealed an overall poor performance by Italian students, especially in the area of problem solving (OECD, 2004).
A closer review of answers to single test items suggested that OECD PISA real-life like questions were quite different from the subject-matter related questions that Italian pupils typically faced in the classroom. For example, they performed comparatively well (often better than students from countries with overall higher scores) in text retrieval tasks, but failed – often by avoiding answering – on more complex questions, especially ones requiring them to draw upon outside knowledge (Bolletta & Pozio, 2008; Caviglia, 2008).
Such findings point to the need of integrating problem-solving practice into the Italian classroom, primarily with the goal of encouraging the students’ willingness and disposition to engage with new problems, a key factor in problem-solving competency (OECD, 2010b, p. 12).
We presented these findings and ideas in a number of teacher education seminaries addressing the theme of ‘new approaches to literacy’, in which we let the teachers struggle both with information problems on the Web and with some of the most difficult items in the OECD PISA literacy surveys. After the last seminar, a teacher of Italian language and her principal jointly asked whether we felt like putting our ideas into practice by organising a study with students in the first year of upper secondary school (14 to 15 years old). Our study was an autonomous activity within a school-wide initiative about ‘learning to learn’ (European Union, 2006), with special focus on reading.
Research study: designing an IPS curriculum
The study involved a class of 23 students in the months between October 2007 and April 2008, for two hours a week, with the exception of three intensive blocks of six hours a week (for a total amount of 50 hours). The activity took place within the curriculum of Italian language teaching and was originally thought of as the first iteration of an educational design experiment (A. L. Brown, 1992; Collins, Joseph, & Bielaczyc, 2004; McKenney & Reeves, 2012). It has not been possible to carry out a second iteration of the study in a comparable setting, even if several assignments from this study were later proposed to older and younger students.
[p. 492]
The study was organised in the following stages:
1. a preliminary phase, in which the students answered a questionnaire and took a test based on questions from the OECD PISA survey (now in OECD, 2009);
2. a first series of IPS sessions, run between October and January, which are the main focus of this article;
3. a half-term evaluation that included an IPS activity, followed by a test of reading comprehension;
4. a second series of IPS sessions, run between January and April, some of which fell under a learning programme on ‘asking questions’ that has been described elsewhere (Davidzon, 2011);
5. a final evaluation based on another set of OECD PISA test items and on the overall school results of the class.
A parallel class of 18 students, which was following a more traditional curriculum with focus on reading, acted as a control group by taking part in the initial and final PISA-inspired tests and in the half-term reading comprehension.
The use of PISA tests in the assessment was primarily meant to observe changes in our students’ willingness to engage with new problems, as reflected for example in the number of answered questions, against the average performance of Italian students. The half-term assessment on reading comprehension was required by the school administration within the school-wide ‘learning how to learn’ initiative, but we were allowed to design it in such a way to include an IPS activity before the reading comprehension.
Preliminary phase
In this phase the students answered a questionnaire about their familiarity with information technology and previous school performance, followed by a test based on questions taken from or inspired by published test items from the OECD PISA survey (now in OECD, 2009) in the areas of reading literacy, problem solving and mathematical literacy (the latter with focus on understanding information in graphical form).
Based on the exit marks from lower secondary school, the experimental class was slightly stronger than the control group, with 10 versus 3 of the students having earned the two higher marks in a scale of four; however, the experimental class had more students (5 vs 3) with the lowest exit mark. Despite these differences, there was no significant difference between the performance of the two classes in the PISA-inspired preliminary test. We transformed the students’ raw scores in the test into standardised scores (mean = 100; SD = 10). The experimental group performed slightly better, with a mean of 101.9 (SE = 1.6) against a mean of 97.3 (SE = 2.9) in the control group. This difference was not significant ($-t(25.7) = 1.35, p > .1$).
We also compared the percentages of correct answers to the PISA questions with the Italian and OECD values (see Table 1), while being aware that students from our geographical area and school type were expected to perform closer to the OECD average than to the Italian one (Abburrà & Landini, 2005; OECD, 2013, pp. 48–49). Both groups’ results roughly followed the already mentioned pattern of Italian students in PISA 2000 and 2003, with good performances on easy items and poor performances – with many omitted answers – on the more difficult ones.
[p. 493]
Table 1. Correct answers to preliminary text based on PISA or PISA-like questions.
| Question | Experimental \((n = 23)\) | Control \((n = 17)\) | ITA | OECD |
|---|---|---|---|---|
| Transit System Q1 (full + partial credit) | 33.3% | 35.2% | 30% | 37% |
| INSPIRED BY Robberies – Q1 (full + partial) | 8.4% | 5.9% | – | – |
| Cinema Outing Q1 (full) | 70.8% | 82.4% | 55% | 56% |
| Cinema Outing Q2 | 79.2% | 58.8% | 66% | 68% |
| INSPIRED BY _TEST SCORES Q1 | 70.8% | 52.9% | – | – |
| EXPORTS Q1 | 95.8% | 70.6% | 68% | 79% |
| EXPORTS Q2 | 33.3% | 35.3% | 39% | 48% |
| Runners – Q1 | 83.3% | 68.8% | 87% | 77% |
| Runners – Q4 | 95.5% | 87.5% | 85% | 74% |
| Runners – Q5 | 95.8% | 81.2% | 86% | 78% |
Source: OECD (2009).
Organising a typical IPS session
Class activities were organised according to this sequence: (1) introduction to the information problem to be solved (10 minutes); (2) IPS activity in the computer lab with 1 PC per student (90 minutes); (3) discussion of the results (10–15 minutes).
The class teacher requested that the students worked alone, with occasional assistance from teacher and experiment leaders, in order to encourage them to be individually active and to make it easier to monitor their progress over time. Students never received suggestions for the websites they were expected to visit: they started from a problem and, after the allotted time spent searching the Web, reading and reflecting, they had to hand in a written answer containing a solution, plus an explanation of the search path and of the students’ reasons for justifying their answer.
Two experiment leaders (Francesco Caviglia and Irith Davidzon) ran the activity and observed the students by looking over their shoulders or watching their computer screens remotely from a teacher computer. They also collected lists of the searches done by the students and silent screen-recordings of the students’ activities in two tasks, one at the beginning and one towards the end of the study.
The following sections present in more detail the two main threads in our study: (a) finding/using information and (b) evaluating information. These two goals are closely connected in the real world, and so both dimensions were often intertwined in classroom practice. However, the two goals also involve different skills and dispositions, therefore we describe separately the design of the study in the two areas.
Web searching
At the beginning of the study, all the students (n = 23) declared themselves to be adequately familiar with the notion of ‘searching on the Web’; 21 answered in a questionnaire that they were able to find information on the Web by themselves, while the remaining two – the only ones who did not have a computer at home – were confident that they could do it with some help (we reproduced question Q5-g in the 2006 OECD PISA student questionnaire on ICT familiarity; see OECD, 2005). [p. 494] As it soon became clear by looking at their search strings in the first activities, very few of the students had a notion of how to widen or narrow their search results by using more generic or specific search strings, while a few of them even wrote questions in natural language.
However misplaced, the students’ confidence in their computer skills allowed us to assign them information problems from the very beginning, providing them with enough practice to appreciate, in the following sessions, direct instruction about search engines and search strategies (a ‘time for telling’ instructional policy, as suggested in Schwartz & Bransford, 1998). After about five weeks, most students were able to formulate a reasonably adequate query, and we kept coaching the ones who were still struggling, until they reached an acceptable level in about eight weeks. The type of problems that the students had to solve is synthesised by a few examples in Table 2.
Table 2. A summary of classroom activities focusing on searching and using information.
|
Task |
Main focus |
|---|---|
|
Using the Web, the students have to identify the place where a picture has been taken, by looking for clues (e.g. a flag, a grave) |
|
|
Using the Web, the student should solve problems similar to the following:
|
|
|
Extremely difficult crosswords, only solvable with the help of web searches |
|
It is also worth mentioning that, during the first month, we awarded only positive marks to those students who had performed successfully. Later on, the students received at least one mark every month, based on cumulative performance and progress, plus extra good marks for a brilliant performance in a task. Weaker students received extra coaching and personalised goals to help them formulate better search strings. As an overall policy, we encouraged the students to take risks and rewarded ‘new insights’ rather than ‘correct solutions’. We insisted that ‘the journey was more important than the destination’ and rewarded, accordingly, those students who had spent time on websites that helped them to better understand the problem, although this typically delayed reaching a solution. The marks, together with the teacher’s excellent reputation among older students and colleagues, helped in the first month to persuade some students – and probably their parents – that the study was a serious school activity, although different from typical school routine.
[p. 495]
At the end of the first term (that is, after three months), all the students – including those without a computer at home – had at least one sufficient mark in ‘finding information’.
Evaluating information
The project’s thread on ‘evaluation of trustworthiness’ had multiple aims. On the one hand, we wished to make the students aware that information (not just on the Web) can be inaccurate and even intentionally misleading. On the other hand, we intended to show how the Web does offer the opportunity to check factual information and make an educated guess about the reliability of a source. Finally, we had the ambition of counteracting a conspiracy-theory attitude which is based on undiscriminating mistrust towards public authorities and mainstream media, underlying a number of theories which are relatively widespread in Italy and elsewhere (‘List of Conspiracy Theories’, 2014; in Italy a hoax-buster service is run by Attivissimo, n.d.).
To achieve this goal, our students were presented with a range of problems that required assessing the trustworthiness of a piece of information.
We started with a made-up appeal calling for the release of a journalist who had been sentenced to years in prison in Germany for criticising his government’s economic policy. The students were asked whether it was a good idea to forward the appeal to other people who cared for freedom of opinion. All our students overlooked some important textual cues as to the text’s intrinsic unreliability (e.g. the indefinite date) and only one student eventually mentioned, ‘Nobody is jailed in Germany for criticising the government.’ At the same time, the students used all their ingenuity to check on the Web for ‘facts’ mentioned in the appeal and did find some implausibility (e.g. the journalist would have started working as a war reporter at 14).
However, more than a few students nurtured serious misconceptions about what can be found on the Web. Some suggested that otherwise plausible information about the journalist was false because they did not find confirmation on the Web, but nobody mentioned that it was unlikely not to find traces on the Web of a public campaign in defence of the imprisoned journalist. In the follow-up discussion with the classroom, we pointed out that the students had indeed some basic grasp of the potentiality of the Web but some erroneously believed the Web to be omniscient, while many were not confident enough in their ability to find information that ought to have been online. This activity, and the lively ensuing discussion, made the students aware of their own naivety about the Web, but we also conveyed the message that some knowledge of the world, combined with close reading of texts and intelligent use of the Web, does make it possible to determine whether a particular text is likely to be a hoax.
[p. 496]
In the following weeks, the students had to check the trustworthiness of a number of ‘real-world’ texts (see Table 3): email scams with forged links and websites, especially ones designed to infect computers or steal sensitive information (as in Delfino, 2013); a few strange-but-true stories; urban legends or technological wonders (e.g. the Biowashball™, which at the time of our study had been endorsed by a popular public figure; see Attivissimo, 2008); pieces of news that did have a basis of truth, but required a better understanding of the question (e.g. the killing of pilot whales in the Faroe Islands, its moral defensibility and whether Denmark has anything to do with it, as suggested in a widespread chain letter); open questions with conflicting answers (e.g. the influence of the lunar cycle on people and animals; see below in the next section).
|
Table 3. A summary of classroom activities focused on evaluating information. |
|
|---|---|
|
Problem |
Main focus |
|
‘Would you forward this appeal in defence of a journalist in jail?’Analysis of an appeal invented by the teachers |
|
|
‘Would you act according to this email?’Examples based on true campaigns, chain letters, spam and fraud |
|
|
‘Is it true that…?’Questions based on urban legends, hoaxes and ‘strange-but-true’ stories |
|
|
‘Would you buy a Biowashball™ to wash your clothes and help the environment?’Examining the endorsement of a product by a public figure and ensuring criticism |
|
|
‘Would you forward the following chain letter accusing Denmark of being responsible for the killing of pilot whales in the Faroe Islands?’Understanding the practice of whaling in the Faroe Islands, and the Faroe Islands’ connection to Denmark |
|
The ‘evaluation of information’ thread in the study was a success, at least with regard to the students’ interest and engagement. They discussed some of the texts with their parents and felt clearly empowered by discovering, for example, that they could easily dismiss some widespread urban legends.
Did the students really become better at evaluating sources? The results of each of the activities summarised in Table 3 would require a detailed analysis, since evaluating the results of each student was more a matter of understanding the process of her or his investigation, rather than the evaluating the final answer or the form of their queries. However, all the students eventually earned positive marks and we observed – during individual coaching and plenary discussion – that all of them had become less naive about information, and aware that sources need to be checked. We will examine some important limitations of their critical competences further below, in the Discussion.
Half-way assessment: some expected and unexpected results
Since reading was one of the focuses of our project, we designed a mid-term test of reading comprehension for our students plus the control group, which had followed a more traditional paper-based curriculum. We wished to verify whether our web-based activities had fostered a habit of focusing on the general meaning of the text, while we did not expect that our activities could lead to any special improvement in reading understanding at sentence level.
[p. 497]
The problem, the tasks and the phases
Our assessment activity had a leading question: ‘Do you believe that the lunar cycle has an influence on frequency of births, human emotion, animal aggression and hair growth?’
To address this question, we organised the activity in different phases:
- Brainstorming and initial questionnaire: the same leading question was presented to our students and to the control group. Both groups had a short brainstorming session in their classrooms, and then each student answered a questionnaire about her/his beliefs in the effects of the moon.
- Reading and searching for information: after the brainstorming, the two groups were exposed to different sources of information, in order to acquire some knowledge about the leading question.
The control group read in the classroom, with help from the teacher, an article from New Scientist discussing the subject (Vines, 2001). Vines examines some contradictory results of scientific investigations regarding the alleged influences of the moon and points out that for every study that finds a correlation, there’s another that doesn’t.
Our experimental study group went instead to the computer lab, with the task of finding out more about the influence of the lunar cycle on at least one phenomenon among emotions in humans, aggression in animals, birth frequency or hair growth in humans. Although nearly all of the students reached an answer, most of them noticed that both sceptics and believers were widely represented on the Web, also among researchers, and that it was not easy to find a convincing answer. As a homework reading assignment, the experimental group also received the article by Vines.
- Reading comprehension: in the following lesson, we assigned to both groups an exercise that addressed both general understanding of the article and detailed explanation of chosen sentences in Vines’s text.
- Final questionnaire: a few days later, after presenting to each classroom the results of the reading comprehension test, we again asked the students to fill out a questionnaire about their beliefs concerning the influences of the lunar cycle. We made it clear that their answers to the questionnaire would not be graded; we were just curious about their current beliefs.
Reading understanding versus students’ beliefs
In the questionnaire delivered after the brainstorming, a majority of students in both groups – with slightly more sceptics in the control group – believed that the lunar cycle had some sort of influence on these phenomena. With beliefs measured as a raw score of 0–10 points (with 10 meaning more scepticism), the experimental group had a mean of 5.91 (SE = .43) and the control group a mean of 6.94 (SE = .34); this difference was not significant (–t(35.9) = –1.86, p > 0.5) (see details of the student answers in Table 4).
[p. 498]
Table 4. Students’ beliefs before and after the test.
— | | | | | | | | | | — | — | — | — | — | — | — | — | — |
Pre-test | Experimental group | | Control group | | Post-test | Experimental group | | Control group | |
| | % | | % | | | % | | % |
Yes/probably yes | 17 | 73.9% | 12 | 66.7% | Yes/probably yes | 1 | 4.3% | 12 | 66.7% |
No/probably no | 5 | 21.7% | 4 | 22.2% | No/probably no | 18 | 78.3% | 5 | 27.8% |
No idea | 1 | 4.3% | 2 | 11.1% | It is uncertain | 4 | 17.4% | 1 | 5.6% |
Total | 23 | 100.0% | 18 | 100.0% | Total | 23 | 100.0% | 18 | 100.0% |
— | | | | | | | | | | — | — | — | — | — | — | — | — | — |
Pre-test | Experimental group | | Control group | | Post-test | Experimental group | | Control group | |
| % | | % | | | % | | % | |
Yes/probably yes | 10 | 43.5% | 3 | 16.7% | Yes/probably yes | 4 | 17.4% | 11 | 61.1% |
No/probably no | 9 | 39.1% | 10 | 55.6% | No/probably no | 15 | 65.2% | 6 | 33.3% |
No idea | 4 | 17.4% | 4 | 22.2% | It is uncertain | 3 | 13.0% | 1 | 5.6% |
| | | 1 | 5.6% | missing answer | 1 | 4.3% | – | |
Total | 23 | 100.0% | 18 | 100.0% | Total | 23 | 100.0% | 18 | 100.0% |
— | | | | | — | — | — | — | — | — | — | — | — |
Pre-test | Experimental group | | Control group | | Post-test | Experimental group | | Control group | |
| | % | | % | | | % | | % |
Yes/probably yes | 16 | 69.6% | 4 | 22.2% | Yes/probably yes | 2 | 8.7% | 6 | 33.3% |
No/probably no | 1 | 4.3% | 10 | 55.6% | No/probably no | 16 | 69.6% | 10 | 55.6% |
No idea | 6 | 26.1% | 3 | 16.7% | It is uncertain | 5 | 21.7% | 2 | 11.1% |
| | | 1 | 5.6% | missing answer | | | – | |
Total | 23 | 100.0% | 18 | 100.0% | Total | 23 | 100.0% | 18 | 100.0% |
[p. 499]
In the reading comprehension exercise, we expected that our experimental group would show a comparable or even better performance in overall understanding of the text, while the control group was possibly stronger in understanding details that might have been highlighted by the teacher. In reality, the two groups answered similarly to questions on general understanding and to the ones about comprehension of specific passages: with raw scores transformed into values with an overall mean of 100 and SD = 10, the experimental group had a mean of 100.9 (SE = 2.0), and the control group a mean of 98.6 (SE = 2.5). This difference was not significant (–t(38) = .7, p > .1).
A striking difference emerged in the final questionnaire about the students’ beliefs. We compared within and between the groups the students’ initial and final beliefs. To this purpose, we graded the answers according to their compatibility with the sceptic views expressed or implied by Vines. The result was a raw score for each student – from 0 to 10 points – where 10 corresponded to full compatibility with Vines’s views. In the final survey the students in the experimental group averaged 8.55 points (SE = .24), while their average in the initial survey had been 5.91 (SE = .43). This difference was significant at the 1% level of significance (t(21) = –5.47, p < .01); it also represented a large-sized effect (r = .76). The control group showed instead no significant change in their views about the effects of the moon: their average score had been 6.94 points (SE = .34) before and 6.31 (SE = .63) after reading Vines. The difference was not significant (–t(15) = 1.11, p > .05).
In other words, the students in the experimental group revised their beliefs and ended up being rather sceptical about the influences of the lunar cycle, even on the question about hair growth, which was not explicitly mentioned by Vines; these students correctly inferred that Vines’s sceptical attitude probably would apply to this question as well. In other words, for the students in the experimental group, Vines helped make sense of the conflicting results that they had found on the Web. For the control group, instead Vines was simply expressing an opinion that did not add to their knowledge of the world. When we expressed our surprise at the fact that the students in the control group did not change their initial beliefs, some even changing them in opposition to Vines’s article, a student explained that ‘well, we did read and understand the article, but we still believe that the moon influences a lot of things’.
Table 4 summarises pre- and post-test answers in three questions about the influence of the moon.
To better visualise the difference between the two groups, we also categorised the students into three groups: Sceptics (who basically agreed with Vines’s sceptic stance about the influence of the moon on human and animal behaviour and physiology), Possibilists (who thought the moon might have some influence) and Believers (who believed in the influence of the moon, and therefore disagreed with Vines), as presented in Table 5. The category of Possibilists presumably covers different cases: for example, students who grasped the general idea expressed in the article, but had strong previous convictions and were still in doubt about one specific area of influence; and students who had understood the single issues described in the article, but were reluctant to extend the conclusions to other areas.
[p. 500]
Table 5. Are the students’ final beliefs aligned with Vines’ article?
| Experimental group (n = 23) | Control group (n = 18) | |
|---|---|---|
| Sceptics (agree with the article) | 13 (56.5%) | 4 (22.2%) |
| Possibilists (partially agree with the article) | 9 (39.1%) | 3 (16.7%) |
| Believers (disagree with the article) | 1 (4.3%) | 11 (61.1%) |
Treating beliefs as test results may seem unfair, but we argue that measuring how much the students had approached Vines’s explanation can be considered a tentative way of measuring the students’ preparation for future learning (Bransford & Schwartz, 1999) in relation to this type of question.
Great caution should be exercised when comparing the results from the two groups, since they had gone through different learning processes and may have been exposed to influences, at school or at home, that we were not able to observe. Moreover, some differences may have resulted from the amount of work (measured in either time spent or total words read) done by the two groups.
However, it is striking to observe how a large majority of the students in the experimental group perceived the explanatory text earnestly as a tool for restructuring their beliefs, while most students in the control group considered the text as a school-only thing, with no relevance to their own understanding of the world. In other words, the two groups demonstrated different dispositions towards learning from a text and learning as a problem-solving activity, engaging with open-ended questions.
Final evaluation
We report and discuss in this section the results of our attempt to evaluate whether sustained engagement with IPS activities had an influence on our students’ performance and attitude towards learning. To answer our research question, we looked – as we did in the preliminary stage of the study – at the students’ answers to a few tests taken from the PISA survey. Subsequently, with input from the class teacher, we looked at the class’ overall performance during the school year.
Answering PISA tests
We submitted the final test to our students and to the control group, and transformed the raw scores of tests into standardised values with an overall mean of 100 and SD = 10. The experimental group performed one standard deviation better than the control group, with a mean of 104.4 (SE = 1.7) against 93.9 (SE = 2.0). This difference was significant at the 1% level of significance (t(38) = 3.8, p < .005); it also represented a large-sized effect (r = .52).
We also compared our students’ performance with each question with PISA data and noticed that the experimental group also did better than the OECD average in 12 out of 14 questions (Table 6).
We observed with special attention the test item Plan International 5.2 from the OECD PISA 2000 survey (Kirsch et al., 2002, pp. 69–72; OECD, 2009, p. 27; R099Q04B in the PISA 2000 Test Item Compendium retrieved from OECD, n.d.-a).
[p. 501]
Table 6. Percentage of correct answers in the final tests. OECD average and Italian results are based on OECD (2009) and (marked with *) on the Test Item Compendia for PISA 2000 and 2003 (OECD, n.d.-a, n.d.-b).
| Experimental | Control | ITA | OECD | |
|---|---|---|---|---|
| Plan International – Q4B (full + partial)* | 39.0% | 17.6% | 18.5% | 17.8% |
| Robberies – Q1 (full credit)* | 26.1% | 11.8% | 3.4% | 13.6% |
| Robberies – Q1 (full + partial)* | 82.3% | 65.2% | 25.2% | 39.4% |
| TEST SCORES Q1 | 69.6% | 29.4% | 16% | 32% |
| Support for president (full)* | 47.8% | 29.4% | 37% | 36% |
| Flu – Q2 | 100% | 76.5% | 70% | 70% |
| Flu – Q3 | 69.6% | 52.9% | 39% | 44% |
| Flu – Q4 | 78.3% | 58.8% | 58% | 53% |
| Flu – Q5 (full credit) | 73.9% | 41.2% | 19% | 31% |
| Flu – Q6 | 43.5% | 41.2% | 33% | 45% |
| Walking – Q1 (full + partial) | 86.9% | 47.0% | 43.2% | 54.6% |
| Growing Up – Q1 | 52.2% | 76.5% | 60% | 61% |
| Growing Up – Q3 | 69.6% | 29.4% | 41% | 46% |
| Growing Up – Q2 (full)* | 60.9% | 41.2% | 40.5% | 55.2% |
| Holiday Q1* | 69.6% | 41.2% | 45.8% | 46.3% |
This challenging question required the student to imagine possible reasons behind one peculiarly low value in a table. The answer could not be inferred from the test item itself; it could only be answered by making a hypothesis based on one’s knowledge of the world. The Plan International question, which had challenged several teachers during our seminars, was therefore in our view a significant test to verify whether more of our students had increased their confidence in their own judgement. In the experimental group, 9 out of 23 students (39%) gave a correct or acceptable answer to this question, against only 3 out of 18 acceptable answers (18%) in the control group. In addition, only five students (21.7%) in the experimental group failed to answer, against over the half of the control group. Indeed, the experimental group not only outperformed the control group, they also did better than the OECD and Italian average (see details in Table 7).
Overall school results
The students who took part in our study apparently developed a positive attitude toward studying at school. With the exception of one student, who moved to a less challenging school after the first few months (and was not included in the statistics),
Table 7. Compared results on the PISA 2000 test item Plan International, question 2.
| Experimental group | Control group | Italy | OECD average | |
|---|---|---|---|---|
| Omitted answer | 5 | 9 | 40.3% | 29.7% |
| 21.7% | 52.9% | |||
| No credit | 9 | 5 | 40.8% | 52.5% |
| 39.1% | 29.4% | |||
| Partial credit | 6 | 3 | 16.0% | 14.5% |
| 26.1% | 17.6% | |||
| Full credit | 3 | 0 | 2.5% | 3.3% |
| 13.0% | .0% | |||
| Total | 23 | 17 | 100% | 100% |
| 100% | 100% |
Discussion
This session is mainly devoted to discussing the above-mentioned results that seem almost too good to be true. While we believe that this study has been a move in the right direction, other observations point to some shortcomings and risks in our approach. Finally, a note discusses some changes in technology and in the use of information technology and computers that should be taken into consideration in designing IPS activities today, six years after the original study took place.
Attitudes towards learning
Some caveats are necessary when we compare – in the final PISA-based tests – the experimental group with the control group, or with the average performance of the Italian students. Both groups had met questions similar to two of the final PISA test items – Robberies and Test score – in the preliminary tests six months earlier, which may explain the good results with these two test items, compared with OECD and Italian students. The students in the experimental group may have wished to reward the teacher and the experimenters with a good final performance, while the control group felt less involved. Moreover, we agree with an anonymous reviewer who suggested that the students in the experimental group, by taking part in the IPS activities, had implicitly benefited from instruction in test answering – even in the form of advice to answer every question – which the control group and the other Italian students in the PISA survey lacked. We suspect indeed that a growing use of PISA-like tests in the Italian lower secondary school may be one reason behind the better performance of Italian students in the 2012 edition of the PISA survey, compared with the 2000 and 2003 editions (INVALSI, 2013). However, it is still an open question why in OECD PISA 2012 the Italian students performed significantly better in the area of problem solving than would be expected from their performance in the other areas (OECD, 2014a, 2014b; Rozzi, 2014).
Even the remarkable overall school result – all the students promoted to the following year – was in part influenced by the teacher who decided to start the experiment, and therefore less significant. On the other hand, we observed how some students who had been admitted to upper secondary school with a low mark performed well in the experiment. Their success with IPS may have influenced their self-esteem positively and, therefore, their performance in other subject areas.
Another positive change could be observed soon after the first lessons, when the students stopped cheating by looking at each other’s screens or surreptitiously asking other fellow students for help. Students started instead looking for a solution themselves or openly asking the teacher for help with search queries and the reliability of sources. The teacher’s charisma, or even the fact that the students were at the beginning of a school cycle and therefore more eager to become involved in school practice, might have contributed to their engagement.
At any rate, the results from both the half-way and final evaluation are consistent with our observations about the engagement of the students throughout the study. This suggests that the web-based IPS activities that we proposed to our experimental ==[p. 503]== group had some transfer value in terms of preparation for future learning (Bransford & Schwartz, 1999). In particular, most of the students seemed to have developed some notion that learning about a topic may involve finding and understanding a reliable source, and also becoming interested in the problem. This notion may have only grown to an embryonic stage – we discuss below the depth of the students’ awareness – but our students did slowly develop a habit of reacting to an information problem with some confidence that it was possible to find a solution or better understand the problem. Students’ persistency increased; they tried hard to find solutions to the assignments; on one occasion, after struggling in the computer lab with a difficult crossword, a few students finished it at home, although this had not been assigned as homework, and proudly presented the result to the teacher.
At the same time, although all students became better at searching information and more than fulfilled our initial goals, the practice of solving information problems on the Web seemed to enhance some differences between them, with a clear advantage for those who were more curious, more tolerant of conflicting results, more reflective or simply better readers. We discuss below how, with most students, our IPS activities fell short of promoting deeper, reflective learning.
Finding versus understanding a solution
On several occasions, we found it difficult to ascertain whether the students had understood a solution or simply found and reported it.
In a previous investigation with a group of Italian educated adults who were asked to explain a puzzling phenomenon, only one subject engaged primarily in constructing the solution rather than finding a ready-made one (Caviglia & Ferraris, 2006; similar results in Selwyn, 2005 and Walraven, Brand-Gruwel, & Boshuizen, 2009). Unsurprisingly, the students in this study also showed the same pattern of behaviour. When they tried to understand whether the lunar cycle really influenced the frequency of births, all but one student looked at competing web pages and tried to figure out which one was more trustworthy, with little interest for the rationales behind competing claims. Only that one especially curious student showed an interest in the method behind the claim and stayed for a long time on one single website in which a maverick statistician explained how he had collected dates of birth from a public register and tried to detect some patterns. In general, when a student displayed a mature attitude towards autonomous knowledge construction, this represented the exception rather than the rule.
Indeed, we suspect that the learning setting designed by us did not encourage reflection. In spite of our policy of rewarding students who slowed down and came up with original observations, the intrinsic element of competition among the students for being the first to find a solution to the problems was an obstacle to reflection.
Evaluating trustworthiness
The students’ inclination towards finding rather than understanding a solution was especially observable with information problems that required evaluating the trustworthiness of a source.
In the PISA 2009 assessment of digital reading among 15-year-old students, the highest level of proficiency was defined as the ability ‘to locate, analyse and ==[p. 504]== critically evaluate information, related to an unfamiliar context, in the presence of ambiguity’ (OECD, 2011, p. 46). At this level, the students can ‘generate criteria to evaluate the text’, while the task ‘may require navigation across multiple sites without explicit direction’ (p. 46). Many of the tasks that we proposed throughout our study focused on the presence of ambiguity and fell within this category, and our students did become better at solving these problems, as became clear throughout the study and the final activities, in which we recorded what happened on the computer screen. But did our students really learn how to generate criteria to evaluate the text?
At the time of our study, no test item from the OECD PISA survey on digital literacy (which was carried out in 2009) had been published. Today, we would include in the final tests the PISA field trial test items presented by Lumley and Mendelovits (2012). However, these tests only provide a partial view of how students proceed when they evaluate information trustworthiness. We chose therefore in this paragraph to discuss in broader terms our observations of how our students approached this class of problems, rather than the results of specific assignments.
The students in the experimental group developed a confident (maybe overconfident) attitude towards information problems that required evaluating trustworthiness. They felt empowered when they were confronted with questions – from real chain letters to assumptions about the influences of the moon – about which they ended up knowing more than their fellow students in other classes, and probably more than their parents (see Birdsong & Freitas, 2012 for a similar project aimed at adult learners). Experiencing first-hand that the Web is full of all sorts of nonsense, our students developed a habit of approaching information on websites with caution. However, when evaluating the trustworthiness of a message, or comparing connecting claims, their primary attitude was to look for an authoritative source that could tell them what was the right answer. For example, our students quickly learnt – without any help by the teachers – that an Italian ‘hoax-buster’ website had embarked on the mission to check and challenge dubious information, from homemade hoaxes to full-scale conspiracy theories (e.g. about 9/11 or so-called chemtrails, see ‘Chemtrail Conspiracy Theory’, 2015). When our students could not find an authority to act as judge, they tried to evaluate the website’s trustworthiness based on cues like the seriousness of the layout or, at best, the website’s affiliation. However, in spite of our efforts, the students showed little interest in understanding the psychological mechanisms on which misleading information often feeds, or identifying internal cues that ought to have activated the reader’s skepticism in the first place. Indeed, while only one student consistently showed a mature attitude towards generating criteria to evaluate the text, the others seemed – to varying degrees – far more superficial, although we are inclined to believe that a majority of them would have answered adequately in the PISA tests on evaluating trustworthiness analysed by Lumley and Mendelovits (2012).
But were our students ready for the next step of being confronted with a real-life, high-stakes problem? Our project, and indeed our mandate in this study, was limited to simple information problems. At the same time, towards the end of the experimentation we felt that many students were ready for more in-depth investigation. Meanwhile, we resisted the temptation to propose high-stakes problems within the study, such as the perceived danger of inoculation, then heatedly discussed on the Web (see Godlee, Smith, & Marcovitch, 2011): the students’ growing proficiency with the Web, combined with their preference for ready-made answers, risked ==[p. 505]== encouraging overconfidence and superficiality, and we did not want to promote such attitudes. For this reason, throughout the rest of the study, IPS activities remained relatively simple although we tried to encourage the students to reflect on how expectations and emotional involvement could make people easier prey for misinformation (Weinrich, 2000). As suggested by a helpful anonymous reviewer, this reflection would benefit from hands-on activities aimed at exposing widespread but faulty reasoning patterns such as ‘uncritical inference’ (Haney, 1955; Taylor, Rudolph, & Foldy, 2008).
Reading
Reading online, under the pressure of finding a solution, had a negative impact on the students’ reading comprehension. A few weeks into the study, our students had become better at skimming web pages and search results. However, they were often inaccurate when reading in detail. We repeatedly observed how some students managed to locate a relevant page only to misread its content, typically under the influence of their own beliefs and expectations. In other words, the habit of anticipating the contents of a text based on a few cues – which the students had developed when skimming a text to assess its relevance – turned into a bad habit when precise understanding was required. The reading device may have influenced the reading performance, although each student used a (then) decent 15-inch screen with a 1024 x 768 resolution. A comparative analysis of reading performances on paper, computer screens and e-readers suggests indeed that traditional computer screens are poor reading devices on which users read more slowly and with less accuracy (Mangen, Walgermo, & Brønnik, 2013; Noyes & Garland, 2008). We tried to minimise this factor by printing out the information problems and any lengthy text that all students were expected to read at the initial stage of an assignment; moreover, the teacher worked on in-depth reading in parallel with our study. However, if we had the opportunity to design a new ‘learning how to learn’ project, we would consider balancing IPS activities with reading-in-depth elements, possibly as complementary elements within the same activity, as was the case with the assignment about the influences of the lunar cycle.
Designing IPS activities today
As mentioned above, learning how to formulate query was a relatively straightforward process for our students. However, some evolutions and changes in the technologies and use of the Web should be taken into account in designing IPS activities today, six years after the original study.
In 2008, the same search string from the same national location yielded the same results from the Google search engine. This is no longer the case: since our study, the Google search engine has undergone several changes. Search results returned by Google are influenced by the user’s and the browser’s search history and location (Google Official Blog, 2009) and are now increasingly attempting to understand the context and the goal of a search string (Hull, 2013). At the time of our study, we were adamant that the students were better off telling Google what to search for instead of having Google guess what they wished. Today, Google has somehow become better at guessing, but the price is less control from the user. Therefore we would today consider alternative search engines or turning off personalised search in ==[p. 506]== an educational setting, in order to obtain the same results with the same search string from different computers in the classroom, and also to allow for ‘cognitive dissonance’ in search results (Bradley, 2009).
As for the setting of our classroom practice, we observed and coached our students, as mentioned, by looking over their shoulders (in person or through a remote connection to their screen) and analysing their search logs afterwards. Today, we would observe and coach the students primarily with help from tools for online collaboration (e.g. GoogleDrive). Moreover, while we still believe it is important for each student to master the technicalities of web searching by herself or himself, we would strongly consider adding a session of peer review on the students’ answers, in which the students would give one another feedback via comments and discussions on shared texts. These tools for collaborative writing could ideally be customised with functions for extracting and analysing the history of the students’ revisions and conversations.
Finally, we observed in the years following the study how easily a good IPS activity may become obsolete, as soon as a well-explained solution makes its way to the first page of search results, thereby compromising the whole process of discovery. Using open questions on recent, hot topics – as we also practised during our study – can be a way of challenging the students to explore uncharted terrain, but this requires putting effort into work that, by design, is doomed soon to become less interesting or relevant. Even worse, this approach risks encouraging presentism in its worst sense of narrow interest in ephemeral contemporary events (Rushkoff, 2013).
Conclusions and directions for future work
The most striking outcome of our experiment was the rise in the students’ self-confidence, autonomy and persistence. Whatever the reasons behind their engagement, the students perceived our IPS activities as empowering, and ended up appropriating technical skills in searching the Web as well as a basic awareness that new sources should be taken with caution. This experience therefore represents a case in which ICT was integrated in formal learning in a way that made school relevant for the students, and promoted inclusion of a few students who initially perceived themselves as ‘weaker’, thereby meeting requirements of relevance, equity and inclusion (Selwyn, 2011; Underwood & Dillon, 2011; Wei & Hindman, 2011).
Our study so far confirms in part our research hypothesis: IPS activities can play a role in fostering the development of a disposition to learning. At the same time, the typical patterns of behaviour that we observed suggested that a learning setting built around search engines and information problems can improve students’ competence in using the Web and to solve a certain class of problems, but can also easily worsen bad habits like looking for ready-made answers and superficial reading. Moreover, while IPS activities helped changing hierarchies based on some students’ previous school history, they also widened the gap between a few more brilliant students and the rest of the classroom. A major challenge for future work on IPS – and more generally on educational technology – is therefore how to build on the growth of those few learners who fully exploit the potential of the digital media to raise the ambition and performance of their peers.
Overall, this study has brought some insight into our initial research question on ‘how to best foster IPS skills in teenager learners’. Challenging problems, a combination of intrinsic and extrinsic rewards plus a combination of monitoring and ==[p. 507]== coaching the students’ efforts have proven effective. The Web is a fast-changing environment and some information problems become rapidly unusable because they are too easy or simply uninteresting, therefore we still consider it a worthwhile direction for future work to design new IPS problems for students of various ages and levels of maturity, as well as to refine the way in which we build learning settings around these problems.
At the same time, IPS activities like the ones presented in this article seem best suited for initiating inquiries that can be further developed by other means – observation, reading, hands-on manipulation, discussion – that more explicitly encourage deep learning. Therefore, the technology for searching the Web is today so easily accessible that the development of IPS competences need no longer be confined to a computer lab, as in the case of our study. We believe therefore that a second major challenge is to design curricula and learning units that build on information problems as challenging and motivating points of departure.
Acknowledgments
The study described in this article was sponsored by the Italian National Research Council, Institute for Educational Technology, Genoa, and directed by Maria Ferraris. Our warmest thanks to Maria for inspiration and support throughout the project. We are also grateful to Giorgia Campodonico, the class teacher; Benedetto Montanari, the school principal at the higher secondary school ‘Ettore Majorana’, Genoa; and Alexia Delfino for supporting us with statistical analysis. Finally, a special thanks to the editor and two anonymous reviewers for their critical and constructive comments.
Notes on contributors
Francesco Caviglia, PhD, at the Faculty of Arts, Aarhus University, is Associate Professor at the Department for Aesthetics and Communication, Aarhus University. He worked in Italy as a secondary school teacher and researcher in educational technology for the Institute for Educational Technology of the Italian National Research Council. His research interests focus on literacy, educational technology and teacher education.
Manuela Delfino is a secondary school teacher of humanities. She has a PhD in ‘Language, Culture, and Information and Communication Technology’ (University of Genoa) and a BA in Humanities (University of Pisa). She also works as a researcher in the field of educational technology. Her major interests include digital literacy, teacher training and distance education, and her current research is focused on approaches to teaching and learning humanities in schools.
References
Abburrà, L., & Landini, S. (2005). As good as the others. Northern Italian students and their peers in other European regions. Turin: Istituto Ricerche Economico Sociali del Piemonte.
Attivissimo, P. (2008, November 28). Biowashball è una palla? [Is Biowashball a lie?] Il Disinformatico. Retrieved from http://attivissimo.blogspot.it/2008/11/biowashball-una-palla.html ==[p. 508]== Attivissimo, P. (n.d.). Il Disinformatico. Website. Retrieved from http://attivissimo.blogspot.it/
Bellanca, J., & Brandt, R. (Eds.). (2010). 21st century skills: Rethinking how students learn. Bloomington, IN: Solution Tree.
Bereiter, C., & Scardamalia, M. (2005). Technology and literacies: From print literacy to dialogic literacy. In N. Bascia, A. Cumming, A. Datnow, K. Leithwood, & D. Livingstone (Eds.), International handbook of educational policy (pp. 749–761). Dordrecht: Springer.
Birdsong, L., & Freitas, J. (2012). Helping the non-scholar scholar: Information literacy for lifelong learners. Library Trends, 60, 588–610.
Bolletta, R., & Pozio, S. (2008). La competenza matematica in PISA e nelle indagini internazionali [Math competences in OECD PISA and other international surveys]. In M. T. Siniscalco, R. Bolletta, M. Mayer, & S. Pozio (Eds.), Le valutazioni italiane e la scuola italiana (pp. 181–275). Bologna: Zanichelli.
Bradley, T. (2009, December 7). Google personalized results could be bad for search. PC World. Retrieved from http://www.networkworld.com/news/2009/120709-google-personalized-results-could-be.html
Bransford, J. D., & Schwartz, D. L. (1999). Rethinking transfer: A simple proposal with multiple implications. Review of Research in Education, 24, 61–100.
Brown, A. L. (1992). Design experiments: Theoretical and methodological challenges in creating complex interventions. Journal of the Learning Sciences, 2, 141–178.
Brown, J. S. (2000). Growing up digital: How the Web changes work, education, and the ways people learn. Change: The Magazine of Higher Learning, 32(2), 11–20.
Bruce, B. C., (2001). The inquiry page: A collaboratory for curricular innovation. Learning Technology Newsletter, 3(1).
Bruce, B. C., & Bishop, A. P. (2002). Using the Web to support inquiry-based literacy development. Journal of Adolescent & Adult Literacy, 45, 706–714.
Burbules, N. C. (2000). Aporias, webs, and passages: Doubt as an opportunity to learn. Curriculum Inquiry, 30, 171–187.
Burbules, N. C., & Berk, R. (1999). Critical thinking and critical pedagogy: Relations, differences, and limits. In T. S. Popkewitz, & L. Fendler (Eds.), Critical theories in education (pp. 45–65). New York, NY: Routledge.
Caviglia, F. (2002). Lie detecting as a step towards critical literacy. L1 – Educational Studies in Language and Literature, 2, 179–220.
Caviglia, F. (2008). Competenze trasversali e discipline: Osservazioni a margine delle indagini OCSE-PISA [Competences across disciplines: Some observations on the OECD PISA surveys]. In B. D’amore, & S. Sbaragli (Eds.), Didattica della matematica e azioni d’aula. Atti del convegno Incontri con la Matematica, 22, Castel San Pietro Terme (Bo), novembre 2008 (pp. 201–204). Bologna: Pitagora.
Caviglia, F., & Ferraris, M. (2006, December). Web-searching for learning: Observing proficient web users working out an information problem. In Kinshuk, D. G. Sampson, J. M. Spector, & P. Isaias (Eds.), Proceedings of the IADIS International Conference on Cognition and Exploratory Learning in Digital Age (CELDA 2006), Barcelona (pp. 440–442). Lisbon: International Association for Development of the Information Society.
Caviglia, F., & Ferraris, M. (2008). The Web as a learning environment: Focus on the content vs focus on the search process. In M. Kendall, & B. Samways (Eds.), Learning to live in the knowledge society: IFIP 20th World Computer Congress (pp. 175–178). Boston, MA: Springer.
Cheeseman, S., Sumsion, J., & Press, F. (2014). Infants of the knowledge economy: The ambition of the Australian government’s early years learning framework. Pedagogy, Culture & Society, 22, 405–424.
‘Chemtrail Conspiracy Theory’. (2015). Retrieved from http://en.wikipedia.org/wiki/Chemtrail_conspiracy_theory
Childs, A., Sorensen, P., & Twidle, J. (2011). Using the Internet in science teaching? Issues and challenges for initial teacher education. Technology, Pedagogy and Education, 20, 143–160.
Collins, A., & Halverson, R. (2009). Rethinking education in the age of technology: The digital revolution and schooling in America. New York, NY: Teachers College.
Collins, A., Joseph, D., & Bielaczyc, K. (2004). Design research: Theoretical and methodological issues. The Journal of the Learning Sciences, 13, 15–42.
[p. 509]
Cunningham, D. J., Arici, A., Schreiber, J., & Lee, K. (2002). Navigating the World Wide Web: The role of abductive reasoning. International Journal of Applied Semiotics, 3(2), 39–58.
Davidzon, I. (2011). Information problem solving e competenze interrogative: il web come ambiente di apprendimento [Information problem solving and interrogative competences: The web as a learning environment]. Unpublished doctoral dissertation, Università Ca’ Foscari, Venice.
Davies, A., Fidler, D., & Gorbis, M. (2011). Future work skills 2020. Palo Alto, CA: Institute for the Future for the University of Phoenix Research Institute.
Dee Fink, L. (2013). Creating significant learning experiences: An integrated approach to designing college courses. San Francisco, CA: John Wiley & Sons.
Delfino, M. (2013). Don’t trash your spam! Reasoning on spam as a way to train critical thinking. In P. M. Pumilia-Gnarini, E. Favaron, E. Pacetti, J. Bishop, & L. Guerra (Eds.), Handbook of research on didactic strategies and technologies for education: Incorporating advancements (pp. 803–809). Hershey, PA: Information Science Reference.
Dennett, D. C. (2013). Intuition pumps and other tools for thinking. New York, NY: Norton.
Dodge, B. (1997). Some thoughts about WebQuests. Retrieved from http://webquest.sdsu.edu/about_webquests.html
Edelson, D. C., Gordin, D. N., & Pea, R. D. (1999). Addressing the challenges of inquiry-based learning through technology and curriculum design. Journal of the Learning Sciences, 8, 391–450.
European Union. (2006, December 30). Recommendation of the European Parliament and of the Council of 18 December 2006 on key competences for lifelong learning. Official Journal of the European Union. Retrieved from http://eur-lex.europa.eu/legal-content/EN/TXT/?uri=celex:32006H0962
Fairclough, N., & Wodak, R. (1997). Critical discourse analysis. In T. A. Van Dijk (Ed.), Discourse as social interaction (pp. 258–283). London: Sage.
Ferraris, M. (2003). Navigare sul WWW a scuola. Ma per andare dove? [Navigating the Web at school: Where are we going?]. TD – Tecnologie Didattiche, 28, 29–41.
Godlee, F., Smith, J., & Marcovitch, H. (2011). Wakefield’s article linking MMR vaccine and autism was fraudulent. BMJ, 342, c7452.
Google Official Blog. (2009, December 4). Personalized search for everyone. Retrieved from http://googleblog.blogspot.com/2009/12/personalized-search-for-everyone.html
Halpern, D. F. (1998). Teaching critical thinking for transfer across domains. Dispositions, skills, structure training, and metacognitive monitoring. American Psychologist, 53, 449–455.
Haney, W. V. (1955). The uncritical inference test. San Francisco, CA: International Society for General Semantics.
Hargittai, E. (2010). Digital na(t)ives? Variation in internet skills and uses among members of the ‘Net Generation’. Sociological Inquiry, 80, 92–113.
Hargittai, E., Fullerton, L., Menchen-Trevino, E., & Thomas, K. Y. (2010). Trust online: Young adults’ evaluation of web content. International Journal of Communication, 4, 468–494.
Hillesund, T. (2010). Digital reading spaces: How expert readers handle books, the Web and electronic paper. First Monday, 15(4). Retrieved from http://firstmonday.org/ojs/index.php/fm/article/view/2762
Hull, J. (2013). Google Hummingbird: Where no search has gone before. Wired – Innovation Insights. Retrieved from http://www.wired.com/2013/10/google-hummingbird-where-no-search-has-gone-before/
Inquiry Page. (2010). Inquiry page. Learning begins with questions. Retrieved from http://www.cii.illinois.edu/InquiryPage/
INVALSI. (2013). OCSE PISA 2012. Rapporto nazionale [OECD PISA 2012. Italian national report]. Retrieved from http://www.invalsi.it/invalsi/ri/pisa2012/rappnaz/Rapporto_NAZIONALE_OCSE_PISA2012.pdf
Jenkins, H., Purushotma, R., Weigel, M., Clinton, K., & Robison, A. J. (2009). Confronting the challenges of participatory culture: Media education for the 21st century. Cambridge, MA: MIT.
[p. 510]
Kirsch, I., De Jong, J., LaFontaine, D., McQueen, J., Mendelovits, J., & Monseur, C. (2002). Reading for change: Performance and engagement across countries: Results of PISA 2000. Paris: Organisation for Economic Co-operation and Development. Retrieved from http://www.oecd.org/edu/school/programmeforinternationalstudentassessmentpisa/33690904.pdf
Kuiper, E., Volman, M., & Terwel, J. (2005). The Web as an information resource in K–12 education: Strategies for supporting students in searching and processing information. Review of Educational Research, 75, 285–328.
Leu, D. J., Jr, Kinzer, C. K., Coiro, J., & Cammack, D. W. (2004). Toward a theory of new literacies emerging from the Internet and other information and communication technologies. In R. B. Ruddell, & N. Unrau (Eds.), Theoretical models and processes of reading (5th ed.). (pp. 1570–1613). Newark, DE: International Reading Association.
‘List of Conspiracy Theories’. (2014). Retrieved from http://en.wikipedia.org/wiki/List_of_conspiracy_theories
Lumley, T., & Mendelovits, J. (2012, April). How well do young people deal with contradictory and unreliable information on line? What the PISA digital reading assessment tells us. Paper presented at the Annual Conference of the American Educational Research Association, Vancouver, BC. Retrieved from http://research.acer.edu.au/pisa/3/
Mangen, A., Walgermo, B. R., & Brønnick, K. (2013). Reading linear texts on paper versus computer screen: Effects on reading comprehension. International Journal of Educational Research, 58, 61–68.
Mayer, R. E., & Wittrock, R. C. (2006). Problem solving. In P. A. Alexander & P. H. Winne (Eds.), Handbook of educational psychology (2nd ed.). (pp. 287–304). Mahwah, NJ: Erlbaum.
McKenney, S., & Reeves, T. C. (2012). Conducting educational research design. London: Routledge.
McKenzie, J. (1997). A questioning toolkit. From Now On: The Educational Technology Journal, 7(3). Retrieved from http://questioning.org/Q7/toolkit.html
Ministry of Education (MIUR). (2008). Scuola: aumentano i non promossi. Risultati scrutini scuole secondarie superiori dopo le verifiche di fine agosto [School: More students are rejected. Final results in higher secondary school after the Summer session]. Press release. Retrieved from http://srvapl.istruzione.it/ministro/comunicati/2008_miur/150908bis.shtml
Ministry of Education (MIUR). (2013, February 5). Indicazioni nazionali per il curricolo della scuola dell’infanzia e del primo ciclo d’istruzione [Curricular national recommendations]. Gazzetta Ufficiale della Repubblica Italiana, 30.
Moss, C. M. (2002). Finding balance: The vices of our ‘versus’. FirstMonday, 7(1). Retrieved from http://firstmonday.org/ojs/index.php/fm/article/view/924/846
Noyes, J. M., & Garland, K. J. (2008). Computer-vs. paper-based tasks: Are they equivalent? Ergonomics, 51(9), 1352–1375.
Organisation for Economic Co-operation and Development (OECD). (2004). Problem solving for tomorrow’s world. First measures of cross curricular competencies from PISA 2003. Paris: Author.
Organisation for Economic Co-operation and Development (OECD). (2005). ICT familiarity component for the student questionnaire. PISA 2006. Main study. Retrieved from http://pisa2006.acer.edu.au/downloads/PISA06_ICT_questionnaire.pdf
Organisation for Economic Co-operation and Development (OECD). (2009). Take the test. Sample questions from OECD’s PISA assessments. Paris: Author.
Organisation for Economic Co-operation and Development (OECD). (2010a). PISA 2009 framework: Key competencies in reading, mathematics and science. Paris: Author.
Organisation for Economic Co-operation and Development (OECD). (2010b). PISA 2012 field trial problem solving framework. Retrieved from http://www.oecd.org/pisa/pisaproducts/46962005.pdf
Organisation for Economic Co-operation and Development (OECD) (2011). PISA 2009 results: Students on line: Digital technologies and performance. Paris: Author.
Organisation for Economic Co-operation and Development (OECD) (2012). Literacy, numeracy and problem solving in technology-rich environments: Framework for the OECD survey of adult skills. Paris: Author.
[p. 511]
Organisation for Economic Co-operation and Development (OECD) (2013). PISA 2012 results: What students know and can do: Student performance in mathematics, reading and science, Vol. I. Paris: Author.
Organisation for Economic Co-operation and Development (OECD). (2014a). PISA 2012 results: Creative problem solving: Students’ skills in taking real-life problems. Paris: Author.
Organisation for Economic Co-operation and Development (OECD). (2014b). Are 15-year-olds creative problem-solvers? PISA in Focus, 38(04). Retrieved from http://www.oecd.org/pisa/pisaproducts/pisainfocus/pisa-in-focus-n38-%28eng%29-final.pdf
Organisation for Economic Co-operation and Development (OECD). (n.d.-a). Database – PISA 2000. Retrieved from http://pisa2000.acer.edu.au/downloads.php
Organisation for Economic Co-operation and Development (OECD). (n.d.-b). Database – PISA 2003. Retrieved from http://pisa2003.acer.edu.au/downloads.php
Ott, N., & Meurers, D. (2010). Information retrieval for education: Making search engines language aware. Themes in Science and Technology Education, 3, 9–30.
Paavola, S., Engeström, R., & Hakkarainen, K. (2012). The trialogical approach as a new form of mediation. In A. Moen, A. I. Mørch, & S. Paavola (Eds.), Collaborative knowledge creation: Practices, tools, concepts (pp. 1–14). Rotterdam: Sense.
Rozzi, F. (2014, April 8). OCSE-PISA 2012: I quindicenni e il ‘problem solving’ [15-year-olds and problem solving]. Adiscuola. Retrieved from http://www.adiscuola.it/adiw_brevi/?p=11710
Rushkoff, D. (2013). Present shock: When everything happens now. New York, NY: Penguin.
Scardamalia, M., & Bereiter, C. (2006). Knowledge building: Theory, pedagogy and technology. In K. Sawyer (Ed.), Cambridge handbook of the learning sciences (pp. 97–118). New York, NY: Cambridge University.
Schwartz, D. L., & Bransford, J. D. (1998). A time for telling. Cognition and Instruction, 16, 475–523.
Selwyn, N. (2005). Reflexivity and technology in adult learning. Seminar.net – Media, technology and lifelong learning, 1(1). Retrieved from http://seminar.net/images/stories/vol1_2005/NeilSelwyn-ReflexityAndTechnology.pdf
Selwyn, N. (2011). Education and technology: Key issues and debates. London: Continuum.
Simone, R. (2002). La terza fase: Forme di sapere che stiamo perdendo [The third phase: forms of knowledge that we are losing]. Bari: Laterza.
Simone, R. (2012). Presi nella rete. La mente ai tempi del web [Caught in the net. The mind in the age of the Web]. Milan: Garzanti.
Taylor, S., Rudolph, J., & Foldy, G. (2008). Teaching reflective practice in the action science/action inquiry tradition: Key stages, concepts and practices. In P. Reason, & H. Bradbury (Eds.), The SAGE handbook of action research: Participative inquiry and practice (pp. 656–668). Thousand Oaks, CA: SAGE.
Thomas, D., & Brown, J. S. (2011). A new culture of learning: Cultivating the imagination for a world of constant change. Lexington, KY: CreateSpace.
Underwood, J., & Dillon, G. (2011). Chasing dreams and recognising realities: Teachers’ responses to ICT. Technology, Pedagogy and Education, 20, 317–330.
Vines, G. (2001). Blame it on the moonlight. New Scientist, 170(2296), 36–39.
Walraven, A., Brand-Gruwel, S., & Boshuizen, H. (2008). Information-problem solving: A review of problems students encounter and instructional solutions. Computers in Human Behavior, 24, 623–648.
Walraven, A., Brand-Gruwel, S., & Boshuizen, H. (2009). How students evaluate information and sources when searching the World Wide Web for information. Computers & Education, 52, 234–246.
Warschauer, M. (1999). Electronic literacies: Language culture and power in online education. Mahwah, MJ: Lawrence Erlbaum Associates.
Warschauer, M. (2002). Reconceptualizing the digital divide. First Monday, 7(7). http://dx.doi.org/10.5210/fm.v7i7.967.
[p. 512]
Wei, L., & Hindman, D. B. (2011). Does the digital divide matter more? Comparing the effects of new media and old media use on the education-based knowledge gap. Mass Communication and Society, 14, 216–235.
Weinrich, H. (2000). Linguistik der Lüge [Linguistics of the lie.] Munich: Beck. (Original work published 1966)
Citation
@article{caviglia2016,
author = {Caviglia, Francesco and Delfino, Manuela},
title = {Foundational Skills and Dispositions for Learning},
journal = {Technology, Pedagogy and Education},
volume = {25},
number = {4},
pages = {487 - 512},
date = {2016},
url = {https://doi.org/10.1080/1475939X.2015.1080756},
doi = {10.1080/1475939X.2015.1080756},
langid = {en},
abstract = {Active participation in the information society requires
the ability to find some order in the chaotic nature of the Web and
not to get lost within the endemic presence of inaccurate,
misleading, biased and false information. This article presents an
approach to Information Problem Solving (IPS) – that is, finding,
understanding and assessing information on the Web – and discusses a
study carried out in an Italian secondary school, using an
experimental and a control group. The study aimed at exploring how
to best foster IPS skills, and observing whether and how IPS
activities could promote the development of more general learning
dispositions and competences. After a period of training with IPS
activities, the experimental group showed different dispositions
towards learning from a text and engaging with open-ended questions.
Despite serious limitations in the depth of analysis, most students
were able to reach acceptable solutions; at the same time, they felt
empowered and developed an embryonic critical attitude on which it
might be possible to build further.}
}