(1 of 3) Measuring the Impact of Technology on Learning

How do we measure the impact of technology used to support learning?

In the first of three posts I reflect on technology’s role in education and how we can measure the impact as we utilise it in our classrooms.

Learning is a complex system, particularly in schools where we try to turbo charge it from 9am to 3pm. The measure of that learning can often be less complex, such as a letter or number to quantify the learning or benchmark the progress since the last check.

Technology has always been part of learning…

Chalk, slate, paper, pens, books and calculators… but we are specifically referring to new technology that has become so ingrained in our lives, therefore schools are grappling with whether and how to embed this technology in learning. If a school gives every child an iPad, will this improve learning? Of course it won’t, any tool is simply a small part of the system but we have become used to measuring impact on learning so much in schools we need to be able to measure the impact of technology on learning but it is crude and dis-ingenuous to measure the impact simply in results. However, schools can spend heavily on technology when budgets are shrinking and the investment needs to be justified.

Measuring Impact in Schools

The measures we are used to are

  • Results
  • Lesson Observations
  • Progress indicators

If I were to visit your school and say

  1. “Implementing technology to support learning will improve results.”
  2. “Lessons with technology are better than those without.”
  3. “Learning with technology will improve student progress”

I sincerely hope you show me the door as I almost definitely don’t have evidence to back the statements up. Furthermore, if those silver bullets existed you would already be using them.

However, I do strongly believe that there are some learning experiences that technology can enhance, improve and provide that are not possible without it.

 

wpid-wp-1391683768119.jpg

There are times we have reflected on lessons and note that mini-whiteboards, post-it notes, laminated cards or highlighters could have made the lesson more successful. This is not because it saves paper, looks pretty but because it improves progress, understanding, explanations, discovery, feedback and so on. Hence, I aspire that technology sits less as an entity on its own and becomes ingrained in learning and teaching. There are fantastic tools freely available to teachers that can enhance feedback (The Education Endowment Foundation and Hattie have research to suggest this has a significantly positive impact on learning. For balance you may wish to also read @LearningSpy’s blog post summarising some objections to the use of effect size and this post questioning the statistics in Hattie’s work) to students or facilitate more effective group work or peer review.

If I were to visit your school and say

  1. “Implementing technology saves teachers time.”
  2. “Lessons with technology allow for more personalised learning.”
  3. “Learning with technology will improve student engagement”

I hypothesise that I would be much less likely to be shown the door. Technology can be a distraction, irrelevant or a positive impact on learning and I suggest the measures we use to assess the impact of technology need to be discussed and a clear set of agreed measures brought in to general use. The team at Google for Education tried this with a vote on twitter:

Technology used to enhance personalised learning requires some cultural change in schools as it challenges the teaching styles of many successful teachers, the learning preferences of good children and the digital literacy of both groups. Alongside any implementation of a change to embed technology in learning is a need for training, discussion and a clear vision for learning. The training deficit that I have become aware of as I try to embed technology in learning was highlighted more publicly in late 2015.

On 15 September 2015, The Organisation for Economic Co-operation and Development (OECD) published a report called “Students, Computers and Learning” on technology in the classroom and snippets from it formed rather sensationalist headlines, including this one on the BBC website.

Computers ‘do not improve’ pupil results, says OECD

The report looks at computer usage across a number of countries and compares outcomes. Using OECD’s Programme for International Assessment (PISA) results from 2012 the report identifies “no appreciable improvements in student achievement in reading, mathematics or science in the countries that had invested heavily in ICT for education”.

Do we ‘invest’ in ICT for Education?

Is investment in ICT for education or technology for learning simply monetary? My school has an investors in people plaque in reception and I am pretty sure that doesn’t mean they pay more than other schools. However, the reference to investment in ICT in the OECD report is a quantitative measure. Despite the media headlines the report is detailed and aware of the data it has and has not used when summarising the findings.

Technology can amplify great teaching, but great technology cannot replace poor teaching.”

OECD: Students, Computers and Learning: Making the Connections, 2015

Schools will have to choose the level and focus of investment in technology for learning. Whether that takes the form of devices, training, open discussion, design thinking projects, bring your own device (BYOD) policies or even new learning spaces is a big and difficult decision for school leaders to make. Especially as the impact of these measures are unclear, hence this blog post.

The discussion around technology in schools is often confused,  school leader. There is a distinct difference between school provided technology to support learning and a personal device used by a student. This distinction can be overlooked in the discussion that surfaces via bloggers, newspapers and those with vested interests. An Ofsted spokesperson was quoted in the Times Educational Supplement (TES):

“Pupils bringing personal devices such as laptops or tablets into school can be extremely disruptive and make it difficult for teachers to teach,” an Ofsted spokesperson told TES.

This quote is then translated into this headline:

Ofsted warns against ‘extremely disruptive’ tablets in school

Richard Vaughan – TES December 2015

Here is a response to the TES article, (Ensure you also read a follow up post including Ofsted’s response as the reference to disruptive technology was specifically to do with technology in schools not for the use in learning) including a search through Ofsted reports for references to technology’s impact on learning.

The decision six months ago to equip all students with tablet computers has not been universally welcomed by parents and carers, but the positive impact on students’ learning is obvious. The computers help students to work independently, they give all students equal access to online resources and they provide an excellent communication tool between teachers and students.

Ofsted Report June 2012 – Secondary (School overall judged as Outstanding)

The impact of technology on learning to this inspection was ‘obvious’ so we know it when we see it. The impact measures suggested by this inspector are

  • Independence
  • Equal access
  • Effective communication

Did this school implement tablet computers because they identified weaknesses in these areas? When I work with schools advising them on their use of technology to support learning the first document I request is the school development plan as there is no point developing technology or the use of it in anything that does not support the key focuses of that school. Therefore the measures have to be the same measures schools put in place every day, week term and year.

The OECD report summarises the implications of the findings and suggests that schools are not yet able or ready to embed technology in learning and leverage the potential. I agree, that we are yet to develop a clarity on how we want technology to blend into pedagogy and schools. I feel this provides more support for a drive to agree tangible measures for the impact of technology. What is the problem we are solving? The implications section concluding the OECD report implies the following potential benefits of technology in education:

  • Equality
  • Digital Literacy
  • Teacher and Student Collaboration

I suspect a search through school development plans will find many that want equality, more officially known and closing the gap. A number of approaches can support this and technology, planned and implemented well, can help. However, a poorly executed implementation of technology can bring into sharp focus the inequality between students. We often hear educators present the now familiar concept that “we are preparing our students for jobs that don’t exist yet” (if I had a penny…) and therefore the use of technology in schools is necessary to prepare them for the world they are going into. However, I didn’t have any such preparation and I think I am doing fine. Technology is already allowing teachers to share and collaborate and we have dumped millions of mediocre to poor resources onto our blogs, social media and resource sites.

What would your measures of impact be?

In part 2 of this 3 parts blog series I want to focus on the tool versus the teacher.

 

Comminucake – Immersion

Our design thinking project, focussed on how we communicate across our 16 school Multi-Academy Trust (MAT) has begun… with a scheduled meeting!

“Meetings of peers sitting around a table rarely lead to collaborative actions towards solving a problem.”

Ewan McIntosh – How to come up with great ideas, 2014.

Indeed, not everyone could make it despite the enticing title “Communicake”; because where do we communicate best… over cake! Not that I am making conclusions at this early stage of the design thinking process, of course.

Project Team

A project team, consisting of at least one representative from each department of the central team, met two weeks ago to begin immersing ourselves in communication. Each team member received a notebook for them to record ideas and bugs related to communication. These observations will be key to the immersion phase as we gather a detailed collection of how communication currently works in our organisation. Not, how we think it works, or how it should work but how it does work day to day.

Immersion Tools

The tools for the immersion will be:

  • Notebooks – Ideas wallet and bugs list
  • Communicake board – Pin board by the coffee and cakes in our office
  • Interviews – It is vital we speak to as many employees across our MAT as possible

First Meeting

wp-1453758918881.jpg

The conversation was productive at beginning to consider the wide scope of communication. Some key questions emerged in-between preferences for face to face over electronic tools for communication. The questions were:

  • How many emails are sent across our organisation everyday?
  • How do private company central offices work?
  • Has every member of the MAT central team visited every school?
  • How many schools have a busy staffroom?
  • Where is our staffroom (one was created the day after)
  • Which schools consider themselves as being “good” at communication?

I have to admit to feeling I had not done a good job of facilitating the beginning of this design thinking process. I am fortunate to have some excellent colleagues who pushed the conversation forward and quickly picked up on the concept of design thinking from my brief description. However, over the week following our project getting started people are highlighting poor or good communication where it may have been overlooked before. For instance, someone has been asked to look at decorating our new office and we ended up discussing whether we should consider communication before deciding what goes on the walls. Hence the person given the task of purchasing decorations is joining us for the next gathering (can’t being myself to call it a meeting). Maybe this design thinking is as effective as I have read?

Next…

  • Review Ideas and Bugs
  • Identify next level of detail required in immersion
  • Interview techniques and tools

 

Making a Dent – The Project Team

After an inspirational two days in Toulouse and a desire to implement lasting change in my organisation, I have been given the support and permission to start a design thinking process for how we communicate across our Multi-Academy Trust (MAT).

As our MAT grows towards 20 schools we will need to be able to communicate effectively across all schools and via the central team that I work in. It feels like just adding more email accounts would not be the solution, so what is? Far be it for me to answer that question, we have many talented people across our organisation so I hope to facilitate their ideas through a design thinking process.

Create the project team

IMG_20151029_225808

We need to create a project team who will begin this journey that I hope will embed across the entire MAT. It needs to be:

  • Representative of the entire organisation
  • Keen to be part of a process of innovation

The team will need to understand that their daily job descriptions are left at the door for this process

“Innovation does not respect departmental boundaries” McIntosh, Ewan. How to Come Up with Great Ideas. Edinburgh: NoTosh, 2014. Print.

I was going to send an email but my CEO made an astute observation and suggestion. If we are looking for innovative ways to communicate effectively then maybe the recruitment should be communicated in an interesting way… tous chez!

Lets design think this with 100 ideas in 10 minutes:

  1. email with a long explanation of the project
  2. email a video of me talking about the project
  3. email a poster about the project
  4. put up a poster in the office
  5. fly a plane with a banner over the school
  6. have a treasure hunt and project team is the ones who complete it first
  7. send a word by email each day to show poor communication
  8. create a video via a QR code
  9. send everyone a copy of Ewan’s book with a note inside
  10. pin up paper with little tear off slips like you had a uni to get a room-mate
  11. Have a person with a “golf sale” sign while I sit in an office waiting for people to come and have a look
  12. buy advertising on TV during Gogglebox
  13. create a radio advert to broadcast
  14. set off the fire alarm and stand in the playground with a loud-hailer to inform everyone of the project
  15. send a survey where people assess which role they fulfil in a team, then pick the right mixture.
  16. Put numbers underneath everyone’s desk and randomly select people to join the team
  17. Set off carrier pigeons from Cornwall with a message inviting colleagues to join the team
  18. Use the force to make them want to join the team “this is not the method of communication you are looking for”
  19. Offer cake and tea in a meeting room and then announce my project
  20. Send a Google form around where people can
    1. watch a video
    2. identify their role in a team
    3. express an interest in getting involved
  21. Have a meeting (you can tell I am running out of ideas…and ten minutes is up!)

I have demonstrated why one needs a project team to power the ideas along and lead to innovative change. Now to get one…

After various attempts at making a video, animated gifs and other media I decided that we all communicate best over an impromptu piece of cake. Consequently I am going to lay on cake and that will lure people to my invitation to get involved:

Communication Flyer
The actual link is not included, just in case you all wanted to join the team.

First question… Why do we need to communicate any differently?

Enjoy

B Rouse

How might we… Evolve #Edtech

Without talking about tech?

After my enthusiastic and bold message to my boss during Ewan McIntosh’s keynote (thanks to lisibo for the reflections and great sketchnotes!) at Practical Pedagogies in Toulouse I have been preparing to meet with him to start making that dent.

My role is to embed technology in learning across the academy chain, however I am coming to the conclusion this is best achieved by not talking about the technology (or at least as little as possible). Conversations about learning, independent learning, sharing and collaboration have been more effective in moving the use of technology forward in an effective way.

Hence, if my colleagues are creating innovative ideas and looking to make them happen, technology that supports learning will follow.

“How might we raise the aspirations of our children?”

“How might we shared best practice across our schools?”

“How might we assess without levels?”

“How might we connect with other cultures?”

“How might we support our local community?”

These are learning based problems that can have a technology aspect in their solutions but technology is not the solution alone. I often refer to a former colleague who was/is a self-confessed “techno-phobe” and nervous of my #edtech role. However, one lunch in the canteen she idly mentioned whatsapp, which she was merrily using on her new iphone. The technology gave her no worries or concerns…. Why? Because it allowed her to view endless pictures of her beautiful granddaughter. It solved a problem and enhanced her experience of being a new grandmother.

The design thinking process seems to be an ideal tool for identifying problems and developing ideas so I want to try and begin to introduce it to my organisation.

IMG_20151029_224921Part of this process has been re-reading Ewan’s book “How to Come Up with Great Ideas… and actually make them happen”. It is a fascinating combination of Ewan’s experience and a journey through the design thinking process. The aspects I have picked up in particular are ideas about trying to move excellence out of individual classrooms, how meetings do not support innovation and looking to define problems, great problems that can lead to great ideas. I have used a lot of post-its, though this seems to be a big part of design thinking!

The key ideas I want to get across are:

  • Help us immerse in our schools and look at the details (avoiding doing what we have always done)
  • Valuing people and their ideas (design thinking gives us a way to get the best ideas from our colleagues)
  • Make our organisation innovative and focussed on learning (Make the big thing, The big thing)
  • Schools can remain individual and autonomous while having a common language for creating innovation across the academies (A key vision of our Multi-academy Trust, MAT)
  • Subtle was of getting everyone utilising technology effectively (Evolving Edtech maybe?)

It feels fuzzy but I am assured this is how all great ideas should feel at the beginning. However, fuzzy doesn’t always allow people to buy-in.

The practical things I am asking for are

  • A notebook for a team/teams to write a bug list and ideas wallet
  • A room to display the immersion of our core team in our schools
  • Lots of post-its
  • Permission to form a project team that represents out entire organisation

My next post may be my last if I cannot convey the impact I think design thinking could have on our organisation.

Watch this space

Ben

Making a dent in more than a sofa

I spent my birthday in a French Bowling Alley with a group of educators in a retail park outside of Toulouse. If that wasn’t enough the highlight of my 35 years I also went minimally viral for sitting on a squidgy sofa. I also got my phone out during a keynote address to send an email to the CEO of my Multi-Academy Trust. These are a few of the events of a pivotal couple of days in my career…

CRcVScJUkAMyvbb
Created by Mark Shillitoe 2015 https://twitter.com/markshillitoe/status/655015698276687872

The official event was Practical Pedagogies organised by Russell Tarr from International School of Toulouse. This is an event based wgich started as an in-house professional development project that grew and grew to an international gathering of educators to share ideas and spend some quality time learning and developing our practice. I was made aware of it by Cindy Crannell, who lead a session with Mark Stillitoe (he took the sofa picture), and applied to do a session about Marking and Feedback as it was an idea I had been developing that I wanted to try out. I shared a selection of Marking and Feedback techniques that I have used that do not involve pen and paper. Hence the slightly provocative title “Marking and Feedback without coloured pens”. My session was accepted and my schools approved a couple of days in Toulouse.

My session aside the event was a triumph and Russell should be very proud of the positive impact it will have on a range of schools and classrooms Europe wide and probably beyond. Thanks to Miles Berry I have a huge amount to take back to the computing curriculum forum I facilitate across a number of Primary Schools.

If the sessions throughout the two days were not enough the keynote speaker was Ewan McIntosh. His company Notosh has some impressive credentials, working with a range of schools and companies to drive positive change using Design Thinking. Since joining twitter and following Tom Barrett, one of Ewan’s colleagues at Notosh, I have been intrigued by Notosh and their apparent clarity of vision which enables them to deliver in a variety of scenarios.

Ewan referred to Steve Job’s quote of about putting a dent in the universe and soon after that I put a rather large dent in a sofa. I shall resist mixed and laboured metaphors but simply say Ewan had me hooked and I am determined to utilise Design Thinking within my own organisation to gain increased clarity on my aim to embed technology in learning across our Multi-Academy Trust.

“We’re here to put a dent in the universe. Otherwise why else even be here?”

— Steve Jobs

In Ewan’s closing session (I was not the only person welling up during the Venezuelan Youth Orchestra) he asked us all to make the big thing, the big thing and decide on what our objective would be and the strategy and tactics to achieve it. The tweet I created at the time is below.

I have adapted it and will probably continue to do so..

Objective:

Embed technology in outstanding learning across my Mutli-Academy Trust

Strategies:

Ensure a robust and appropriate infrastructure for each school that they can rely on. Build confidence in all staff to be aware of opportunities to enhance learning with technology.

Tactics:

Design Thinking Design thinking is a process that Notosh use in all their work to help organisations identify problems and create ideas and prototype solutions. I have been threatening to use this for a while and Ewan’s keynote was the reminder I needed to get on with it. This blog post and subsequent ones are my reflections of this process which starts now. Check back in to find out how we get on on the journey from making a dent in a sofa to making a dent in the universe…

If you would like to see everyone else’s objectives, strategies and tactics click here. Thanks to Dave Stacey for compiling them via storify.

Enjoy

Ben