Engaging with intelligent technology

 
Annie Grant talked to RIX Research and Media to discover the potential of intelligent technology for learners with the most complex needs.

The young people of Charlton Park Academy in South London are very comfortable working with new technology. The school has a wide range of assistive and interactive technologies, and also houses the Centre for Motor and Associated Communication (CENMAC), which supplies assistive technologies to children right across London.

‘We are always keen to explore the potential of new technology because it is hugely engaging, it can remove barriers and for some young people with learning disabilities it can be everything’, explains CENMAC team leader Kathryn Stowell. ‘When eye-gaze technology first came in, for instance, it meant that, for the first time, our students with profound and multiple learning difficulties (PMLD) could do something without any adult support and really demonstrate their level of understanding.’

Personalising learning

It was the Academy’s vision and enthusiasm for technology that led Gosia Kwiatkowska, Senior Lecturer and Co-director of RIX Research and Media Centre at the University of East London, to approach Charlton Park to help pilot MaTHiSiS (Managing Affective-learning THrough Intelligent atoms and Smart InteractionS) a personalised and adaptable e-learning system, developed by a team from across Europe and funded by the European Union Horizon 2020 Framework Programme.

This cutting-edge technology continually adjusts online learning paths to keep learners motivated. It does this through an algorithm that determines a user’s engagement through combining their performance on e-learning tasks with their emotional state, as indicated by their facial expressions and body movements. Although possible applications of the technology are wide-ranging – as part of the project, MaTHiSiS was piloted with users in mainstream and special schools, industry and careers guidance – the RIX Centre’s involvement focused mainly on the potential of the system for young people with PMLD and ASD.

‘This is the group whose needs are hardest to meet and who will benefit most from the technology because securing engagement is crucial if they are to learn,’ says Ms Kwiatkowska, who was also excited by the possibility of the young people themselves influencing the design. ‘As a group, people with PMLD are so often overlooked because they don’t have a voice, but in this project, they have the opportunity to contribute directly to the design of this technology, maybe not consciously but, nevertheless, they are the pioneers,’ she adds.

The MaTHiSiS learning platform has the capability to be used in conjunction with several different interfaces simultaneously so that teachers can create learning experiences personalised to meet the different needs of their students. ‘You could have a child in one corner of the classroom working on a computer, while students work individually or collaboratively on tablets or a big screen,’ explains Ms Kwiatkowska. Charlton Park, who had purchased a robot, were very keen to explore the possibility of linking the platform with this new hardware. 

But the main attraction of MaTHiSiS for RIX and for Charlton Park Academy was its potential to enable students with PMLD to work more independently, without constant adult oversight. Typically, a child with PMLD will have an adult beside them at all times, assisting with tasks and regulating their emotional state and behaviours by recognising and responding to signs of, pleasure, motivation, frustration or distress. By using a specialist webcam, MaTHiSiS continuously assesses a child’s engagement, assigns an emotional state and then presents learning materials in a way that is most likely to keep them motivated to continue learning. ‘It means that students don’t get bored because material is too easy, or frustrated because it is too difficult,’ explains Ms Kwiatkowska.

Evaluation of the project has indicated that because MaTHiSiS is able to detect emotional states from a range of very subtle indicators, it may be better at assessing engagement than an adult sitting next to the student. But Ms Kwiatkowska is keen to emphasise that the system does not replace the teacher. ‘It potentially increases independence for students by removing the need for an adult to anticipate their responses and keep them on task, because the machine will do that,’ she says.

Technical development

The MaTHiSiS project ran for just over three years, finishing in March 2019. It had three phases:

• Driver – developing the algorithm and the learning materials
• Assisted – RIX and schools piloting and evaluating the technology together, and
• Real-life – Teachers integrating technology into their day-to-day classroom teaching.

In the ‘driver’ phase the project partners worked to develop the algorithm. Across Europe, children’s facial expressions were recorded while they were working on online tasks. Education professionals who knew children really well annotated the video material, indicating points when children’s reactions indicated that they were engaged, motivated, bored or frustrated etc. As the information from hundreds of observations was fed into the algorithm, it was able to learn more and more about the different ways in which young people show engagement.

Teachers also worked closely with RIX during this phase to develop the learning materials to upload to the learning platform. ‘For our target group we were looking at very simple tasks involving cause and effect, matching and basic literacy and numeracy,’ explains Gosia Kwiatkowska.

Assisted implementation

With the platform in place and functioning, the pilot moved on to the second ‘assisted’ phase, in which RIX worked with teachers and teaching assistants at Charlton Park Academy to set up the prototype system and to test its usability, before the final ‘live pilot’ stage. In practice, things didn’t go as smoothly as hoped. ‘Although we could see it had great potential, the learning platform didn’t really work very well when we came to implementing it in school,’ remembers Kathryn Stowell. ‘We had technical issues and the learning materials weren’t really a good fit for the needs of our PMLD students – even though they were simplified, they were still too difficult for children to access – so we moved to testing the system with more able pupils with complex needs.’

Miss Kwiatkowska agrees that things didn’t go as well as planned but points out that it is only through prototypes being piloted in these very early stages that difficulties can be identified and corrected. She applauds Charlton Park Academy’s open-mindedness and willingness to persist with the pilot so that the system could be improved for the future. ‘Early pilots of complex technology do take teachers and students out of their comfort zone. They can disrupt the operation of day-to-day activities and sometimes it’s great for students, and sometimes it’s not,’ she explains. ‘But it was so helpful to be able to feed their comments back into the development process, to improve the functioning of the platform in the future for groups who so often get left out’.                                                                                                                             

Because of technical difficulties, the pilot was never able to proceed to the ‘live’ stage, where teachers integrated the technology into their everyday teaching, without support from RIX. But the system did become more stable as time went on and the school obtained some success in using MaTHiSiS with their new target pupil group, generating some interesting and useful feedback and data for the project.

Increasing independence

A success of the pilot was the algorithm, which seemed to predict pupils’ emotional states accurately, challenging students and moving their learning a little further without triggering negative behaviours. But if the system was really to free students to work more independently, teachers and support staff needed to believe in it and learn to trust it.

In a feedback interview, Claire, a teaching assistant at Charlton Park Academy reported that her concern about the welfare of students tended to make her second guess the technology and jump in when she anticipated that students might begin to struggle. ‘I didn’t trust the computer to be able to assess their level of frustration as well as I could,’ she explains. ‘I knew by their body language when they were becoming anxious and, although you want to encourage students, you don’t want to harm them either.’ 

On the other hand, Anne, an experienced teacher, was more confident to step back and see how well students interacted with technology on their own. ‘There was a temptation to jump in too soon because you were anticipating when they might be getting bored or frustrated,’ she says. ‘But I realised it was OK to hold back a little and I noticed then that system not only stretched them a little bit, but it made them a little more independent. It reduced their reliance on the adult and I thought, well hang on, maybe sometimes it could be the adults that are holding them back.’ 

Gosia Kwiatkowska recognises how decisions that teachers make with the best intentions, can limit opportunities for students. ‘Children with more complex needs come to rely on adult’s ‘protecting’ them from things that may be perceived as too challenging, but MaTHiSiS gives students the chance to make mistakes and to try again and again, without the teacher stepping in, so long as the system assesses that they are still motivated to do so,’ she says. ‘And by allowing students to take those extra steps, MaTHiSiS can challenge teachers’ preconceived ideas about the abilities of students with learning disabilities and reveal capabilities that teachers may not previously have understood or realised they had.’

Engagement and progression

At first students at Charlton Park were a little wary of the new set-up but they soon gained in confidence. And while it was never possible to make the robot work with MaTHiSiS during the pilot, it was used as a reward to encourage them to participate and persevere with the online tasks.

The robot was a key motivator for some children but others surprised staff by their determination to succeed in the online tasks, without an incentive. Gosia Kwiatkowska recalls one student who couldn’t get over a barrier in one of the online tasks. ‘The system kept presenting the same activity over and over again and he was still failing,’ she remembers. ‘We asked if he wanted to stop and play with the robot but he said no. He was motivated and engaged by the activity and the system was recognising that and pushing him to try again, and he felt an enormous sense of achievement when he got through it.’

Students with learning disabilities typically make very small steps in learning progress and the MaTHiSiS system recognises and reports on this, along with data about how much and when a child has remained engaged. Kathryn Stowell recognises the usefulness of this for tracking and reporting on students’ progress and for teachers as they choose learning resources. ‘It added to the information we have to demonstrate students’ cognitive abilities and encourages teachers to think more critically about the resources they use in the classroom, in terms of their ability to engage and motivate students,’ she says.

Looking to the future

After three years the project is now at end and an evaluation has been prepared. Generally, teachers who had used MaTHiSiS were positive about it and despite concerns about technical issues, believed strongly that the system contributed to greater engagement and independence for learners.

Gosia Kwiatkowska hopes that new funding may be available in the future so that the technology can fulfil its potential and, eventually, become a commercial package available for all schools to use.

A major area for development is the appropriateness of the learning materials for target groups, especially those with PMLD. ‘It’s important that the technical teams and practitioners continue to work together so that everyone has a good understanding both of the potential of the technology and the needs of different groups of children, so that the product is really fit-for-purpose,’ explains Kathryn Stowell. Ms Kwiatkowska recognises this challenge. ‘Students with PMLD are unique and we all need to recognise that and take account of it going forward. You cannot develop a ‘PMLD persona’ and develop materials for just that one persona,’ she says.

There are of course other challenges, particularly around the stability and usability of the platform to support individual and collaborative teaching and learning in a busy, inclusive classroom. And staff training will also be needed to address what Ms Stowell describes as ‘a step change in teaching approach’, similar to that required when eye-gaze technology became available for teaching and learning.

Despite the challenges of the project, Kathryn Stowell remains really positive about the potential of MaTHiSiS. ‘We have a long-standing relationship with RIX and it was really exciting to be involved in a project that focused on the needs of young people, especially those with PMLD. It’s really important that their voice is heard when new educational technologies are developed so that they can show the potential they have to learn and achieve.’

This article was originally written for RIX Research & Media.