Explaining computerized English testing in plain English

ɫèAV Languages
a pair of hands typing at a laptop

Research has shown that automated scoring can give more reliable and objective results than human examiners when evaluating a person’s mastery of English. This is because an automated scoring system is impartial, unlike humans, who can be influenced by irrelevant factors such as a test taker’s appearance or body language. Additionally, automated scoring treats regional accents equally, unlike human examiners who may favor accents they are more familiar with. Automated scoring also allows individual features of a spoken or written test question response to be analyzed independent of one another, so that a weakness in one area of language does not affect the scoring of other areas.

was created in response to the demand for a more accurate, objective, secure and relevant test of English. Our automated scoring system is a central feature of the test, and vital to ensuring the delivery of accurate, objective and relevant results – no matter who the test-taker is or where the test is taken.

Development and validation of the scoring system to ensure accuracy

PTE Academic’s automated scoring system was developed after extensive research and field testing. A prototype test was developed and administered to a sample of more than 10,000 test takers from 158 different countries, speaking 126 different native languages. This data was collected and used to train the automated scoring engines for both the written and spoken PTE Academic items.

To do this, multiple trained human markers assess each answer. Those results are used as the training material for machine learning algorithms, similar to those used by systems like Google Search or Apple’s Siri. The model makes initial guesses as to the scores each response should get, then consults the actual scores to see well how it did, adjusts itself in a few directions, then goes through the training set over and over again, adjusting and improving until it arrives at a maximally correct solution – a solution that ideally gets very close to predicting the set of human ratings.

Once trained up and performing at a high level, this model is used as a marking algorithm, able to score new responses just like human markers would. Correlations between scores given by this system and trained human markers are quite high. The standard error of measurement between ɫèAV’s system and a human rater is less than that between one human rater and another – in other words, the machine scores are more accurate than those given by a pair of human raters, because much of the bias and unreliability has been squeezed out of them. In general, you can think of a machine scoring system as one that takes the best stuff out of human ratings, then acts like an idealized human marker.

ɫèAV conducts scoring validation studies to ensure that the machine scores are consistently comparable to ratings given by skilled human raters. Here, a new set of test-taker responses (never seen by the machine) are scored by both human raters and by the automated scoring system. Research has demonstrated that the automated scoring technology underlying PTE Academic produces scores comparable to those obtained from careful human experts. This means that the automated system “acts” like a human rater when assessing test takers’ language skills, but does so with a machine's precision, consistency and objectivity.

Scoring speaking responses with ɫèAV’s Ordinate technology

The spoken portion of PTE Academic is automatically scored using ɫèAV’s Ordinate technology. Ordinate technology results from years of research in speech recognition, statistical modeling, linguistics and testing theory. The technology uses a proprietary speech processing system that is specifically designed to analyze and automatically score speech from fluent and second-language English speakers. The Ordinate scoring system collects hundreds of pieces of information from the test takers’ spoken responses in addition to just the words, such as pace, timing and rhythm, as well as the power of their voice, emphasis, intonation and accuracy of pronunciation. It is trained to recognize even somewhat mispronounced words, and quickly evaluates the content, relevance and coherence of the response. In particular, the meaning of the spoken response is evaluated, making it possible for these models to assess whether or not what was said deserves a high score.

Scoring writing responses with Intelligent Essay Assessor™ (IEA)

The written portion of PTE Academic is scored using the Intelligent Essay Assessor™ (IEA), an automated scoring tool powered by ɫèAV’s state-of-the-art Knowledge Analysis Technologies™ (KAT) engine. Based on more than 20 years of research and development, the KAT engine automatically evaluates the meaning of text, such as an essay written by a student in response to a particular prompt. The KAT engine evaluates writing as accurately as skilled human raters using a proprietary application of the mathematical approach known as Latent Semantic Analysis (LSA). LSA evaluates the meaning of language by analyzing large bodies of relevant text and their meanings. Therefore, using LSA, the KAT engine can understand the meaning of text much like a human.

What aspects of English does PTE Academic assess?

Written scoring

Spoken scoring

  • Word choice
  • Grammar and mechanics
  • Progression of ideas
  • Organization
  • Style, tone
  • Paragraph structure
  • Development, coherence
  • Point of view
  • Task completion
  • Sentence mastery
  • Content
  • Vocabulary
  • Accuracy
  • Pronunciation
  • Intonation
  • Fluency
  • Expressiveness
  • Pragmatics

More blogs from ɫèAV

  • Children sat at desks at computers in a classroom

    21st-century skills and the English language classroom

    By Sara Davila

    Are you teaching in a 21st-century classroom? Chances are, If you are an English educator working in the classroom today, you have already moved well ahead of your peers and colleagues teaching math, science, and good old-fashioned grammar. Now that you know you are a 21st-century teacher, what does that mean? And how do you know if you have moved ahead of the curve to embrace what we call 21st-century skills?

    Actually, "21st-century skills" is a bit of a misnomer. The prized skills of this age have existed in teaching and learning as long as we have been teaching and learning. In a modern-day class, Socrates and Aristotle would feel right at home (although maybe underdressed).

    The phrase itself is meant to imply a classroom ready for the upcoming STEM needs of employment that will allow for innovation, development and significant advances across tech and non-tech industries. Yet, the skills themselves do not imply a highly technological classroom. A modern 21st-century class can be a surprisingly low-budget place.

    It can be summarized by the 4Cs:

    • Communication
    • Critical Thinking
    • Creativity
    • Collaboration

    Reading through this list, you may think, "Hey, those are my classroom goals as an English language teacher!" Finally, the rest of the world has caught up with the modern English language classroom. Of course, when describing these skills, we aren't just talking about teaching English, but skills that can be used to prepare learners for the modern age. This means we want our students to be able to:

    • Perform independently and with groups in a highly technologically advanced atmosphere.
    • Be ready for daily, global interaction.
    • Be capable of adaptive, flexible and creative thinking.
    • Understand how to plan for, build, and include collaboration with peers who are colleagues and experts in the field.

    Students and 21st-century skills

    This goes a bit above and beyond the basics of the walls of the English language classroom. And yet, preparing our students for the 21st century doesn't require a classroom resembling a science fiction movie set. Several teachers have proved that you can embed these skills by utilizing the most important resource available in the classroom.

    Your students.

    Sergio Correra is an inspired young teacher at the Jose Urbina Lopez Primary School on the US Border with Mexico. After a year of teaching uninspired curriculum to disengaged students, he returned to the drawing board. He spent time researching ways to improve student engagement and performance and stumbled across exciting research that could be boiled down to one question: Why? Or rather, getting students to ask the question: "Why?" At the beginning of his next school year, he arranged the desk in a circle, sat his students down and asked: "What do you want to learn about?".

    Using this as the jumping-off point, he encouraged students to ask questions, seek out more information, and find more questions to answer.

    Over the next year, he saw his students' test scores rise, the engagement and enthusiasm improved and he received approval from his principal and fellow educators. With few resources and limited access to technology, he found his students shifting from the lowest testing group in the nation to being ranked among the highest for their performance on standardized tests in the country. One of his students was the highest-performing maths student in the country.

    Mr. Correra was inspired by research and reports based on the work of the Indian educator Sugata Mitra. The principle behind Mr. Mitra's approach is to drive student's curiosity by letting them carry out their own learning. In one of his most famous examples, he walked into a classroom in India with computers loaded with information. He explained to the students, now curious about the big shining boxes that held inside something interesting.

    And then he left the students to it.

    In the course of a year, students had taught themselves everything from English to molecular biology, all without the guidance of a teacher. Rather, they were driven by their natural curiosity, playing off of each other's discoveries to go farther and learn more. Embodying what it means to be self-guided, innovative, collaborative and curious learners.

    Keeping your curriculum up to date

    These students who were given freedom are much more likely to ask questions out of curiosity, motivate themselves and learn without guidance. And while this may be wonderful for learners, this isn't exactly helpful for teachers. To get to the 21st-century skills and inspire motivation, do we have to throw away our syllabus and books and trust only in our learners to motivate themselves?

    Fortunately for those of us who have chosen a career in education, that is not the case. We as educators can take lessons from Mr. Correra and Mr. Mitra and use these as a way to inspire interest and engagement in our own classroom while building these skills in our learners.

    As language teachers, it's a matter of blending the 4Cs more thoughtfully into a student-centered classroom where learners can engage in high-interest content that is relevant, useful, and promotes innovation.

    Take your average prepositions lesson as an example. Even in the best communicative classroom, a teacher may still spend time explaining the rules, setting up the activity and delivering instruction. By applying the 4Cs we can turn this lesson a bit more on its head, making a typical ELL grammar lesson magical.

    For example:

    Collaborate: Start by handing out magazines or picture books. Have the students work together and choose a picture.

    Communication, critical thinking, and creativity: Ask your studentsto work together to create two ways to give directions. One set of directions for a student who is blind. Another set of directions for a student who is deaf.

    Encourage students to think outside the box and think about ways to give directions using a computer, a mobile phone, a television, or a YouTube video. While there may be some L1 use in the classroom, the goal is for the final product to be in English. Stand back and watch your learners go.

    Another way to engage with 21st-century skills using a typical ELL lesson: the "What's your favorite food lesson?" At some point, we have all experienced it.

    Collaborate: In groups, have students create a survey to assess classroom interest in 10 different foods representing different types of meals (breakfast, lunch, dinner, dessert).

    Communication: Once finished, have learners use the information to create a pie or bar graph to communicate the results and determine which meals are the favorite.

    Critical thinking: Have the students compare their answers with answers from other groups. How many differences are there in the reporting? Is the information consistent with the same foods or does it change drastically? Have students compare their results with other teams. Then ask the groups to create a short written or spoken piece to explain how their results differed from other students.

    Creativity: Using the information collected from the class and after analyzing data from other students, have groups work together to create an advertising campaign that will make the foods that students liked least into foods students may like more. For example, if the survey said that most students did not like for breakfast, the group would need to work together to create an advertising campaign to make kim-chi-chigae seem like a tasty choice for breakfast. To do this students should consider what makes certain foods more popular in the class.

    This may require further follow-up interviewing to find out why students like one thing and not another; this information can then be used in the campaign. This lesson may play out over a few days, but in the end, everyone involved will have gotten much more out of the lesson than they had anticipated.

    Both of these examples represent the use of skills in the ELL classroom. Each lesson also embeds, in one way or another, critical STEM skills.

    In the preposition lesson, the students may use engineering and technology to find a better way to give directions. In our favorite foods lesson, students engage with science (and a bit of sociology) and mathematics. Altogether it becomes a rounded classroom experience where teachers have an active role as facilitators and students become inspired, self-guided learners who still manage to work inside of the confines of the curriculum.

    In the end, 21st-century skills, and using them in the classroom is not really about teaching at all. These skills are truly ones that will spell success for our learners in the future, leading them to be capable, Independent and curious individuals.

    Our real challenge as educators is to model a desire to embrace the known, the unknown, and the just plain unknowable. As Alwin Toffler, writer and futurist, put it: "The illiterate of the 21st century will not be those who cannot read and write, but those who cannot learn, unlearn, and relearn."

  • University students sat in a classroom at desks with a teacher speaking to them

    Planning for success with the GSE

    By Sara Davila

    The Global Scale of English (GSE) is the first truly global English language standard.

    It consists of a detailed scale of language ability and learning objectives, forming the foundations of our courses and assessments at ɫèAV English.

    The GSE was developed based on research involving over 6000 language teachers worldwide. The objective was to extend the current descriptor sets to enable the measurement of progression within a CEFR level – and also to address the learning needs of a wider group of students.

    It can be used in conjunction with a current school curriculum and allows teachers to accurately measure their learners’ progress in all four skills of reading, writing, listening and speaking.

    GSE was introduced at the – an English language school run by the University of Toledo in Ohio, USA – with impressive results.

    The American Language Institute

    The Institute provides English courses for students who want to improve their English and prepares students to take the International Student English exam. They offer an intensive language program consisting of 20 hours of classes every week and 40 hours of self-study. This 60-hour week is designed to fast-track students from a lower level of English to a standard which allows them to participate successfully in college courses. There are five course levels offered, from A2+ to B2+ and class sizes average at around 10 students.

    Most students at the Institute are full-time international students planning to attend the University of Toledo once their English language proficiency reaches the required standard. On average, they are between 18 and 20 years old, and enter the language program with a B1 level of English.

    A mission statement

    At the Institute, the main aim of the language courses is to help students develop their English skills to a level that will allow them to integrate successfully into the university community, not just academically but socially. In their own words; “Our ultimate goal isn’t to teach them how to take and pass language tests, but to teach them how to use English and engage themselves with the local communities.”

    So how did the GSE, in conjunction with the Versant test and other ɫèAV products, help to achieve this goal?

    Transitioning to a objectives-based curriculum

    First, the course coordinator Dr Ting Li adopted the GSE for a more detailed approach to the CEFR. She found that the GSE “made the CEFR more manageable because it broke out the levels and outlined CEFR goals into different categories.”

    Next, she replaced the current course materials with NorthStar Speaking & Listening, NorthStar Reading & Writing, and Focus on Grammar. These courses covered the areas taught in the previous curriculum, as well as the three key areas of study; literacy, speaking and listening, and grammar.

    The instructors also began using ɫèAV English Connect, a digital platform for teachers and students.This gave them the flexibility to revise questions and reduce administrative burden due to the automatic grading feature.

    Finally, the Institute started using the Versant English placement test to decide which level students should enter when they first begin studying at the Institute.

    Key findings from the case study

    The new curriculum was a great success. Students, teachers and administrators all found that the courses and assessments, all underpinned by the GSE, made the language learning experience smoother and easier. Once students had completed the highest level of the course and achieved a 3.0 GPA, they were able to transition smoothly into their courses at the University of Toledo.

    The alignment between the NorthStar courses, the grammar study books and the Versant test was informed by the GSE. This meant students didn’t have to sit as many assessments as before, reducing time teachers had to spend setting and marking exams, and allowing them to focus more on supporting learners and the quality of their lessons.

    Dr Li highlighted the following key benefits:

    • The Global Scale of English supports the development of a standardized curriculum and a consistent framework for teaching English
    • The average student GPA was highly related to the University of Toledo’s undergraduate GPA, which indicates that if students do well at the Institute, they will have a successful academic career.
    • There was no group difference between graduates of the Institute and the average University of Toledo student GPA, which indicates that the Institute’s students perform as well as other international students who have been directly admitted to the university.
    • There was no difference between credits earned 2 years into the university program compared with the general student population.

    What’s more, the Institute was recently recognized by the , meaning that the course run by Dr Li is now nationally recognized. Using the GSE to inform the organization of the course curriculum made the accreditation process smoother and easier.

    Working as a team

    One of the main pieces of feedback from Dr Li and the Institute was how helpful they found the ɫèAV representatives, who offered excellent customer support, building a sense of a team between their representatives and the school. This very teamwork helped the Institute to fulfill the ambition in their mission statement. It makes for an inspiring story of how one school used the GSE to transform their curriculum, and achieved their goal of helping students to improve their English and achieve their academic ambitions.

  • a young man sat in a lecture hall with other students behind him

    How the GSE helped Salem State University meet learner needs

    By Sara Davila

    Salem State University is one of the largest and most diverse public teaching universities in Massachusetts. In total, it has about 8,700 students enrolled, 37% of whom are people of color. It also educates 221 international students from 59 different countries – with China, Albania, Brazil, Morocco, Nigeria and Japan among the most represented countries on campus.

    The university runs an intensive English language program. Most students who enrol come from China, Brazil, Albania, Vietnam, and Japan. The program also has a number of part-time English language learners from the local community.

    In 2016, Associate Director Shawn Wolfe and teachers at the American Language and Culture Institute did a review and found that areas for growth included establishing a universal documentation for identifying learner needs, goals and progress.

    “The biggest challenge was that we needed to have a better way of placing students,” Wolfe says. “We also needed to have a way to have our curriculum, our assessment and our student learning outcomes unified.”

    The team lacked programmatic data related to learning gains and outcomes. Additionally, they realized that assessments could be used to inform students about entry requirements at the university and other programs. And that’s where the Global Scale of English (GSE) came in, as a tool which enabled the staff at the American Language and Culture Institute to personalize and diversity their English teaching program to meet learner needs.

    Cultural and linguistic diversity

    David Silva PhD, the Provost and Academic Vice President, highlights the need for this type of personalization when it comes to education.

    “We have to be prepared for an increasing variety of learners and learning contexts. This means we have to make our learning contexts real,” he says. “We have to think about application, and we have to think about how learners will take what they learn and apply it, both in terms of so-called book smarts, but also in terms of soft skills, because they’re so important.”

    Silva makes the point that, as the world gets smaller and technology becomes a bigger part of our lives, we can be anywhere at any time, working with anyone from across the globe. “We need to be prepared,” he says, “for those cultural and linguistic differences that we’re going to face in our day-to-day jobs.”

    The ability to change and adapt

    So how does the curriculum at the American Language and Culture Institute help prepare students for the world of study and work?

    At the Institute, the general review led to the realization that the program needed to be adaptive and flexible. This would provide a balance between general English and academic preparation and would also encompass English for specific purposes (ESP).

    Wolfe says, “The GSE fit with what we were trying to do because it offers three different options; English for academic learners, English for professionals and English for adults, which is another area that we realized we needed to add to our evening program so that we can serve working adults that are English language learners in our community.”

    The English language instructors at the Institute were also impressed with the capabilities of the GSE. Joni Hagigeorges, one of the instructors, found the GSE to be an excellent tool for tracking student progress.

    “What I really like is that you can choose the skill – , listening, speaking – and you’re given the can-do statements, the learning objectives that each student will need to progress to the next level,” she said.

    Wolfe also commented on the GSE Teacher Toolkit and the way that it supports assessment and planning, allowing instructors to get ideas for specific learning objectives for groups or individual students. “It’s enabled us to personalize learning, and it’s changed the way that our teachers are planning their lessons, as well as the way that they are assessing the students.”

    A curriculum that will meet learner needs

    The GSE has allowed the team at the Institute to become more responsive to changing student expectations. The alignment of placement and progress tests to the GSE has allowed instructors to have more input into the courses they are teaching.

    Elizabeth Cullen, an English language instructor at the Institute, said, “The GSE helps us assess the strengths and weaknesses of various textbooks. It has helped us develop a unified curriculum, and a unified assessment mechanism.”

    This unification means that the curriculum can easily be tweaked or redesigned quickly to meet the needs of the students. What’s more, as Elizabeth points out, the students benefit too. “The Global Scale of English provides students with a road map showing them where they are now, where they want to go and how they’re going to get there.”

    Standing out from the crowd

    In this time of global hyper-competition, the challenge for any language program is finding innovative ways to stand out from the crowd while staying true to your identity. At Salem State, the staff found that the GSE was the perfect tool for the modern, data-driven approach to education, inspiring constant inquiry, discussion and innovation. It offers students, instructors and administrators a truly global metric to set and measure goals, and go beyond the ordinary.