Explaining computerized English testing in plain English

ɫèAV Languages
a pair of hands typing at a laptop

Research has shown that automated scoring can give more reliable and objective results than human examiners when evaluating a person’s mastery of English. This is because an automated scoring system is impartial, unlike humans, who can be influenced by irrelevant factors such as a test taker’s appearance or body language. Additionally, automated scoring treats regional accents equally, unlike human examiners who may favor accents they are more familiar with. Automated scoring also allows individual features of a spoken or written test question response to be analyzed independent of one another, so that a weakness in one area of language does not affect the scoring of other areas.

was created in response to the demand for a more accurate, objective, secure and relevant test of English. Our automated scoring system is a central feature of the test, and vital to ensuring the delivery of accurate, objective and relevant results – no matter who the test-taker is or where the test is taken.

Development and validation of the scoring system to ensure accuracy

PTE Academic’s automated scoring system was developed after extensive research and field testing. A prototype test was developed and administered to a sample of more than 10,000 test takers from 158 different countries, speaking 126 different native languages. This data was collected and used to train the automated scoring engines for both the written and spoken PTE Academic items.

To do this, multiple trained human markers assess each answer. Those results are used as the training material for machine learning algorithms, similar to those used by systems like Google Search or Apple’s Siri. The model makes initial guesses as to the scores each response should get, then consults the actual scores to see well how it did, adjusts itself in a few directions, then goes through the training set over and over again, adjusting and improving until it arrives at a maximally correct solution – a solution that ideally gets very close to predicting the set of human ratings.

Once trained up and performing at a high level, this model is used as a marking algorithm, able to score new responses just like human markers would. Correlations between scores given by this system and trained human markers are quite high. The standard error of measurement between ɫèAV’s system and a human rater is less than that between one human rater and another – in other words, the machine scores are more accurate than those given by a pair of human raters, because much of the bias and unreliability has been squeezed out of them. In general, you can think of a machine scoring system as one that takes the best stuff out of human ratings, then acts like an idealized human marker.

ɫèAV conducts scoring validation studies to ensure that the machine scores are consistently comparable to ratings given by skilled human raters. Here, a new set of test-taker responses (never seen by the machine) are scored by both human raters and by the automated scoring system. Research has demonstrated that the automated scoring technology underlying PTE Academic produces scores comparable to those obtained from careful human experts. This means that the automated system “acts” like a human rater when assessing test takers’ language skills, but does so with a machine's precision, consistency and objectivity.

Scoring speaking responses with ɫèAV’s Ordinate technology

The spoken portion of PTE Academic is automatically scored using ɫèAV’s Ordinate technology. Ordinate technology results from years of research in speech recognition, statistical modeling, linguistics and testing theory. The technology uses a proprietary speech processing system that is specifically designed to analyze and automatically score speech from fluent and second-language English speakers. The Ordinate scoring system collects hundreds of pieces of information from the test takers’ spoken responses in addition to just the words, such as pace, timing and rhythm, as well as the power of their voice, emphasis, intonation and accuracy of pronunciation. It is trained to recognize even somewhat mispronounced words, and quickly evaluates the content, relevance and coherence of the response. In particular, the meaning of the spoken response is evaluated, making it possible for these models to assess whether or not what was said deserves a high score.

Scoring writing responses with Intelligent Essay Assessor™ (IEA)

The written portion of PTE Academic is scored using the Intelligent Essay Assessor™ (IEA), an automated scoring tool powered by ɫèAV’s state-of-the-art Knowledge Analysis Technologies™ (KAT) engine. Based on more than 20 years of research and development, the KAT engine automatically evaluates the meaning of text, such as an essay written by a student in response to a particular prompt. The KAT engine evaluates writing as accurately as skilled human raters using a proprietary application of the mathematical approach known as Latent Semantic Analysis (LSA). LSA evaluates the meaning of language by analyzing large bodies of relevant text and their meanings. Therefore, using LSA, the KAT engine can understand the meaning of text much like a human.

What aspects of English does PTE Academic assess?

Written scoring

Spoken scoring
  • Word choice
  • Grammar and mechanics
  • Progression of ideas
  • Organization
  • Style, tone
  • Paragraph structure
  • Development, coherence
  • Point of view
  • Task completion
  • Sentence mastery
  • Content
  • Vocabulary
  • Accuracy
  • Pronunciation
  • Intonation
  • Fluency
  • Expressiveness
  • Pragmatics

More blogs from ɫèAV

  • Children sat at desks in a classroom, one is smiling and looking to the front of the class

    English: the best second language for your child to learn

    By Steffanie Zazulak
    Reading time: 2 minutes

    As adult learners, our very motivation for learning English can sometimes hinder our progress because we are focusing too much on the end result. The informal way in which children learn English – through music, games and fun activities – offers an environment where they can learn and practise without worrying about the importance of it all. This relaxed attitude, in turn, gives them confidence in learning English and sets them up for more opportunities in their academic pursuits and future career options.

    the positive impact bilingualism has on a child’s cognitive development. Catherine Ford, head teacher of Moreton First Prep School, says that children : “Before children become self-conscious they can try out their newly acquired languages without fear of embarrassment”.

    Starting the English learning process at a young age will provide the head start that most parents are keen to give their children in life, education and career. More than 77% of parents who were interviewed as part of said they would consider sending their child to study at a university abroad, which involves studying in English.

    Educational benefits

    The number of students pursuing postgraduate studies overseas continues to rise, reflecting the global nature of education. According to the seeking diverse academic experiences and cultural immersion. One crucial factor in this journey is having the right level of English skills, especially when applying to universities in popular destinations such as the US, UK, and Australia.

    Learning English from a young age provides a solid foundation, enabling students to tackle more complex language skills tailored to their academic goals. Traditional English teaching often emphasizes reading, writing, and grammar, but studying abroad offers a unique opportunity to immerse oneself in an English-speaking culture, enhancing speaking and listening skills.

    Future career benefits

    Mastering English at an early age can be a transformative asset for future career success. English is the lingua franca of business, opening doors to global opportunities and enabling individuals to pursue diverse career paths across borders. As the most widely used language in business worldwide, proficiency in English is a powerful motivator for students aspiring to join global companies.

    Bilingualism is becoming increasingly advantageous in the job market, improving employability and making candidates more appealing to employers. , underscoring the competitive edge that language skills provide.

    Empowering the next generation

    The benefits your children are given by learning English at a young age are invaluable and as they go through life, the possibilities for advancement in their academic and business careers will be wide open. Children are fortunate to have intuitive language learning capabilities from a young age and this is certainly something to capitalize on.

  • Students sat in a university library talking

    Why your US university needs PTE Academic and ɫèAV English Express Test

    By Abi Fordham
    Reading time: 2 minutes

    Higher education is competitive and universities need to attract top talent from around the globe. One way to do this is by offering a range of secure, accurate and trusted proficiency tests. Adding the ɫèAV English Express Test alongside PTE Academic can be a game-changer for both students and admissions teams.

    Meeting diverse student needs

    The ɫèAV English Express Test is a convenient option for students facing time, location, or financial constraints. Unlike traditional tests, it can be taken online, allowing students to complete it from their own homes. This flexibility is particularly beneficial for students in remote areas, who are not close to a test centre.

    Complementing PTE Academic

    While PTE Academic is a truly comprehensive assessment, the ɫèAV English Express Test offers a quicker, at-home option for prospective students.

    Enhancing the admissions process

    For admissions teams, the ɫèAV English Express Test can enhance efficiency and effectiveness. By offering a convenient testing option, universities can attract a broader pool of applicants. The quick turnaround times for results also mean faster decision-making, reducing the waiting period for students.

    Boosting business outcomes

    From a business perspective, integrating the ɫèAV English Express Test can help to increase enrolment numbers and revenue. The enhanced efficiency in the admissions process, through the use of ɫèAV’s trusted Score Reporting Website, can reduce administrative costs and free up resources for other strategic initiatives.

    Additionally, offering flexible testing options can enhance the university's appeal, making it more attractive to prospective students and their families.

    What does this mean for you as an institution?

    Adding the ɫèAV English Express Test alongside PTE Academic is a strategic move that benefits both students and universities. It addresses diverse student needs, complements existing testing options, enhances the admissions process, and boosts business outcomes.

    By embracing this approach, universities can attract top talent and position themselves as leaders in higher education in the US.

  • A teacher stood in front of her class with her students raising their hands

    What is rapid prototyping and how can it apply to the ELT classroom?

    By Nicole Kyriacou
    Reading time: 5 minutes

    Tom Chi is an internet veteran with quite a resumé. His roles have been many and varied – from astrophysical researcher to Fortune 500 consultant and corporate executive, developing new hardware and software products and services.

    He worked on Microsoft Outlook when it was in its infancy, was a major influence in taking Yahoo Search from 0 to 90 million users and is now Head of Product Experience at Google X – Alphabet’s secretive division focused on creating technological innovations for the future. It has produced the self-driving car and Google Glass, and its Project Loon aims to provide internet to every square inch of the earth.

    At Google X, Tom was in a unique position – always having to think five, ten or even more years ahead in order to conceptualize and build the technology of the future. As you might imagine, this is far from an easy task; not only do the ideas have to be original, but they have to meet people’s future needs – something that is not easy to predict.

    So, how does Tom and the others at Google X deliver their vision for the future using today's materials and technology?

    That’s where Rapid Prototyping comes in. It’s a concept that allows teams to experiment, learn and adjust prototypes quickly and cheaply, so that projects (and products) get off the ground. Failure is seen as a starting block and an inevitable part of the learning process. Following his workshop, we are going to look further at rapid prototyping and how it can relate to the ELT classroom.