Explaining computerized English testing in plain English

ɫèAV Languages
a pair of hands typing at a laptop

Research has shown that automated scoring can give more reliable and objective results than human examiners when evaluating a person’s mastery of English. This is because an automated scoring system is impartial, unlike humans, who can be influenced by irrelevant factors such as a test taker’s appearance or body language. Additionally, automated scoring treats regional accents equally, unlike human examiners who may favor accents they are more familiar with. Automated scoring also allows individual features of a spoken or written test question response to be analyzed independent of one another, so that a weakness in one area of language does not affect the scoring of other areas.

was created in response to the demand for a more accurate, objective, secure and relevant test of English. Our automated scoring system is a central feature of the test, and vital to ensuring the delivery of accurate, objective and relevant results – no matter who the test-taker is or where the test is taken.

Development and validation of the scoring system to ensure accuracy

PTE Academic’s automated scoring system was developed after extensive research and field testing. A prototype test was developed and administered to a sample of more than 10,000 test takers from 158 different countries, speaking 126 different native languages. This data was collected and used to train the automated scoring engines for both the written and spoken PTE Academic items.

To do this, multiple trained human markers assess each answer. Those results are used as the training material for machine learning algorithms, similar to those used by systems like Google Search or Apple’s Siri. The model makes initial guesses as to the scores each response should get, then consults the actual scores to see well how it did, adjusts itself in a few directions, then goes through the training set over and over again, adjusting and improving until it arrives at a maximally correct solution – a solution that ideally gets very close to predicting the set of human ratings.

Once trained up and performing at a high level, this model is used as a marking algorithm, able to score new responses just like human markers would. Correlations between scores given by this system and trained human markers are quite high. The standard error of measurement between ɫèAV’s system and a human rater is less than that between one human rater and another – in other words, the machine scores are more accurate than those given by a pair of human raters, because much of the bias and unreliability has been squeezed out of them. In general, you can think of a machine scoring system as one that takes the best stuff out of human ratings, then acts like an idealized human marker.

ɫèAV conducts scoring validation studies to ensure that the machine scores are consistently comparable to ratings given by skilled human raters. Here, a new set of test-taker responses (never seen by the machine) are scored by both human raters and by the automated scoring system. Research has demonstrated that the automated scoring technology underlying PTE Academic produces scores comparable to those obtained from careful human experts. This means that the automated system “acts” like a human rater when assessing test takers’ language skills, but does so with a machine's precision, consistency and objectivity.

Scoring speaking responses with ɫèAV’s Ordinate technology

The spoken portion of PTE Academic is automatically scored using ɫèAV’s Ordinate technology. Ordinate technology results from years of research in speech recognition, statistical modeling, linguistics and testing theory. The technology uses a proprietary speech processing system that is specifically designed to analyze and automatically score speech from fluent and second-language English speakers. The Ordinate scoring system collects hundreds of pieces of information from the test takers’ spoken responses in addition to just the words, such as pace, timing and rhythm, as well as the power of their voice, emphasis, intonation and accuracy of pronunciation. It is trained to recognize even somewhat mispronounced words, and quickly evaluates the content, relevance and coherence of the response. In particular, the meaning of the spoken response is evaluated, making it possible for these models to assess whether or not what was said deserves a high score.

Scoring writing responses with Intelligent Essay Assessor™ (IEA)

The written portion of PTE Academic is scored using the Intelligent Essay Assessor™ (IEA), an automated scoring tool powered by ɫèAV’s state-of-the-art Knowledge Analysis Technologies™ (KAT) engine. Based on more than 20 years of research and development, the KAT engine automatically evaluates the meaning of text, such as an essay written by a student in response to a particular prompt. The KAT engine evaluates writing as accurately as skilled human raters using a proprietary application of the mathematical approach known as Latent Semantic Analysis (LSA). LSA evaluates the meaning of language by analyzing large bodies of relevant text and their meanings. Therefore, using LSA, the KAT engine can understand the meaning of text much like a human.

What aspects of English does PTE Academic assess?

Written scoring

Spoken scoring

  • Word choice
  • Grammar and mechanics
  • Progression of ideas
  • Organization
  • Style, tone
  • Paragraph structure
  • Development, coherence
  • Point of view
  • Task completion
  • Sentence mastery
  • Content
  • Vocabulary
  • Accuracy
  • Pronunciation
  • Intonation
  • Fluency
  • Expressiveness
  • Pragmatics

More blogs from ɫèAV

  • A group of students stood in a classroom high fiving eachother

    How to help students achieve their New Year’s resolutions

    By Nicola Pope
    Reading time: 2 minutes

    2025 is here. As we step into the new year, it's the perfect time to reflect on our recent challenges andachievements. It’s also a good moment to think about the future with optimism andplan our goals. Our students, too, are thinking about their New Year's resolutions.

    As a teacher, you can help them consider how learning English will help them now and in the future. On top of this, you can guide them as they plan their goals and give them useful advice on how to achieve them.

  • People sat in chairs doing various things like working on a laptop; sat in one of those seats is a cartoon robot

    English for employability: What will jobs be like in the future

    By Nicola Pope

    What do driverless car engineers, telemedicine physicians and podcast producers have in common? About 10 years ago . They are representative of a new technology-driven marketplace, which is evolving faster than employers, governments and education institutions can keep up.

    As new jobs appear, others fall by the wayside. Today, it’s estimated that up to with currently available technology. Routine jobs like data entry specialists, proofreaders, and even market research analysts are especially at risk of becoming redundant within the next 5 to 10 years. Globally, that means between 400 and 800 million workers could be displaced by automation technology by 2030, according to McKinsey.

    Moreover, will need to work in areas that do not exist in the current market. The question is, what can we do to prepare learners for a future when we have no idea what jobs they’ll be doing? Mike Mayor and Tim Goodier discuss this uncertain future and explain why English for employability is such a hot topic right now.

    A rising level of English and employer expectations

    Mike Mayor, Director of the Global Scale of English at ɫèAV, explains that while he believes employability has always been a factor in English language education, it has become more important and more of a focus for students looking to enter the workforce.

    “Expectations of employers have risen as proficiency in English language, in general, has risen around the world,” he says. “They’re now looking for more precise skills.”

    Tim Goodier, Head of Academic Development at Eurocentres, agrees. He explains that English language education is primarily about improving communication and soft skills – which is key for the jobs of 2030 and beyond.

    “There’s a convergence of skills training for the workplace and language skills training,” Tim says. “The Common European Framework of Reference (CEFR) has recognized and, in many ways, given a roadmap for looking into how to develop soft skills and skills for employability by fleshing out its existing scheme – especially to look at things like mediation skills.”

    How the Global Scale of English and CEFR have surfaced employability skills

    TheGlobal Scale of English (GSE) is recognizing this increasing prominence of English for employability. Mike explains that it’s doing this “by taking the common European framework and extending it out into language descriptors which are specific for the workplace.”

    In developing a set of learning objectives for professional learners, Mike and his team have given teachers more can-do statements. “They are able to create curricula and lessons around specific business skills,” he says.

    Tim comments that one of the most interesting things about the GSE is that it links can-do statements to key professions, which he explains “is another extension of what these can-do statements can be used for – and viewing competencies as unlocking opportunity.”

    Showing how these skills and competencies relate to the real world of work can be a strong motivating factor for learners.

    He says that teachers need to visualize what success will look like in communication “and then from there develop activities in the classroom that are authentic.” At the same time, he says that activities should be personalized by “using the learners’ own interests and adapting the course as much as possible to their future goals.”

    Preparing students for the future workplace

    Speaking on the role of publishing in English for employability, Mike says:

    “I would say as course book creators we actually incorporate a lot of these skills into our materials, but… I think we could do to push it a little further.”

    In Mike’s view, educators need to do more than teach the skills, they need to raise awareness of their context. In other words why these skills are important and how they will help them in authentic situations both in and out of the work environment.

    Beyond teaching the language itself, he says publishers should be helping teachers ask:

    • Are the students participating fairly in group discussions?
    • Are the students actively listening?
    • Are they interrupting politely?

    These skills “don’t come naturally, and so just to begin raising awareness would be an added value,” he says.

    Future skills: careers in 2030

    In the same way we didn’t know that driverless cars would become a reality 10 years ago, we cannot say with absolute certainty which professions will arise and which will disappear. However, using tools like the GSE teacher toolkit, we can help our students develop the language and soft skills they need to navigate an ever-shifting job market. The future is an exciting place, let’s help our learners prepare themselves!

    Watch the full interview with Mike and Tim below:

  • A group of young people sat together smiling

    How long does it take to learn English?

    By Nicola Pope

    “How long will it take me to learn English?” This is a question we often hear, especially with summer intensive courses just around the corner. Students all over the world want to know how much time and effort it will take them to master a new language.

    Teachers know the answer isn’t as simple as it seems. It depends on many things, such as; how different the second language is from their mother tongue, how old they are, whether they can speak other languages, how much time they will have to study outside the classroom, their motivation and ability to practice.

    The truth is, it takes A LOT of work to become proficient in a new language – and students need to be aware that they need to study independently if they want to progress rapidly.

    Explaining student responsibility

    Becoming truly proficient in a language can take many years. In a study carried out by ɫèAV they found that even for fast learners, it can take as much as 760 hours to enter the B2 CEFR level from <A1.

    Also, most year-round courses are around 100-120 hours per level, (not including homework). So the reality is that it should take approximately 1000 hours to go from A1 to C2.

    However, one of the biggest misconceptions students have is that there is a “fixed route” to language learning and that this is linear – and that time spent studying in class is all that’s required to make the progress they expect. This mistakenly puts the onus on the teacher, rather than the student, which means they may not take responsibility for their own learning.

    While most language learners need great course materials, instruction, correction, and mentorship from their teachers, it’s key that they are motivated to become independent learners. Progress and success comes down to regular practice, feedback and the confidence to make and learn from mistakes. Students must understand this from the outset – so make sure this is a conversation you have with your classes from the very first day.

    Understanding language goals

    It’s also extremely important to understand your students’ language learning goals right away. Some, for example, will want to learn a language for travel purposes and may be happy to reach an elementary or pre-intermediate level of English. Others will want to learn it for work or study purposes and will need to reach a more advanced level. By definition, “learning a new language” will be very different for those two groups of students – and this will affect how you design and deliver your course.

    Therefore, it’s key that you discuss individual learning objectives and then form a plan of how students will meet them. You should also explain that not everyone progresses at the same rate, but that is normal and should not be a cause for frustration.

    In private language schools (PLSs), which offer English for specific purposes (ESP), business English, CLIL, English for Academic Purposes, intensive summer classes, and a range of other courses, it’s even more important to do this well. Correctly managed expectations, well-selected materials, and tailored courses will keep students motivated and help the business thrive.

    Setting and meeting targets

    At an institutional level, schools, PLS’s and even government agencies also need to be aware of the pitfalls of rigid target setting.

    Not only can mishandled targets directly affect learner motivation when they are held back or moved up too quickly, but they also can force educators to “teach to the test”, rather than planning classes and designing courses that meet their students’ needs.

    On the other hand, standardized testing systems help place learners at the right level, set benchmarks and show student progression. Examinations also give students firm objectives to work towards.

    So, at the very least, management and governing authorities should consult with educators before setting broad targets.

    Handling feedback and adapting to individual needs

    Honesty is essential when talking to individual students about their progress (good or bad). It’s hard telling someone that they haven’t achieved the grades they need to move on to the next level, but it’s the right thing to do. Putting a person in a higher level to save their feelings only leads to frustration, demotivation, and self-doubt. Likewise, when a student has done well, praise is good, but you should still be honest about the areas in which they need to improve.

    This is what happens at a successful PLS in Japan who run 1000-hour year-round intensive courses. They get results because they consult their learners in order to understand their goals and focus their courses on developing key communicative skills for professionals. At the same time, they track motivation levels and adjust their courses to ensure the student’s progress is on track to meet their expectations. Of course, this is quite a unique setting, with a very intensive, highly personalized approach, and the school has the advantage of tailor-making courses.

    Using tools to help

    They also used the Global Scale of English (GSE) to help design their curriculum and use the ‘can do’ descriptors to set goals. They then selected Versant assessments (which are mapped to scoring against the GSE) to measure student progress on a monthly basis.

    Educators can emulate their approach. By using tools like these, as well as others, such as the GSE Teacher Toolkit, you can design syllabi, plan classes, place students at the right level and measure individual progress, helping you meet your institution’s targets while supporting your learners to achieve their goals.

    An additional benefit from using the GSE, is that this granular framework breaks down what needs to be learned within a CEFR level. Our courseware, Placement, Progress and high-stakes assessments, like PTE Academic, are already aligned to the GSE. To help accelerate the learner journey, our courseware now features three new levels – A2+, B1+ and B2+. By moving to eight-level courses, it ensures students are able to master the content at a more achievable rate.