Explaining computerized English testing in plain English

色猫AV Languages
a pair of hands typing at a laptop

Research has shown that automated scoring can give more reliable and objective results than human examiners when evaluating a person’s mastery of English. This is because an automated scoring system is impartial, unlike humans, who can be influenced by irrelevant factors such as a test taker’s appearance or body language. Additionally, automated scoring treats regional accents equally, unlike human examiners who may favor accents they are more familiar with. Automated scoring also allows individual features of a spoken or written test question response to be analyzed independent of one another, so that a weakness in one area of language does not affect the scoring of other areas.

was created in response to the demand for a more accurate, objective, secure and relevant test of English. Our automated scoring system is a central feature of the test, and vital to ensuring the delivery of accurate, objective and relevant results – no matter who the test-taker is or where the test is taken.

Development and validation of the scoring system to ensure accuracy

PTE Academic’s automated scoring system was developed after extensive research and field testing. A prototype test was developed and administered to a sample of more than 10,000 test takers from 158 different countries, speaking 126 different native languages. This data was collected and used to train the automated scoring engines for both the written and spoken PTE Academic items.

To do this, multiple trained human markers assess each answer. Those results are used as the training material for machine learning algorithms, similar to those used by systems like Google Search or Apple’s Siri. The model makes initial guesses as to the scores each response should get, then consults the actual scores to see well how it did, adjusts itself in a few directions, then goes through the training set over and over again, adjusting and improving until it arrives at a maximally correct solution – a solution that ideally gets very close to predicting the set of human ratings.

Once trained up and performing at a high level, this model is used as a marking algorithm, able to score new responses just like human markers would. Correlations between scores given by this system and trained human markers are quite high. The standard error of measurement between 色猫AV’s system and a human rater is less than that between one human rater and another – in other words, the machine scores are more accurate than those given by a pair of human raters, because much of the bias and unreliability has been squeezed out of them. In general, you can think of a machine scoring system as one that takes the best stuff out of human ratings, then acts like an idealized human marker.

色猫AV conducts scoring validation studies to ensure that the machine scores are consistently comparable to ratings given by skilled human raters. Here, a new set of test-taker responses (never seen by the machine) are scored by both human raters and by the automated scoring system. Research has demonstrated that the automated scoring technology underlying PTE Academic produces scores comparable to those obtained from careful human experts. This means that the automated system “acts” like a human rater when assessing test takers’ language skills, but does so with a machine's precision, consistency and objectivity.

Scoring speaking responses with 色猫AV’s Ordinate technology

The spoken portion of PTE Academic is automatically scored using 色猫AV’s Ordinate technology. Ordinate technology results from years of research in speech recognition, statistical modeling, linguistics and testing theory. The technology uses a proprietary speech processing system that is specifically designed to analyze and automatically score speech from fluent and second-language English speakers. The Ordinate scoring system collects hundreds of pieces of information from the test takers’ spoken responses in addition to just the words, such as pace, timing and rhythm, as well as the power of their voice, emphasis, intonation and accuracy of pronunciation. It is trained to recognize even somewhat mispronounced words, and quickly evaluates the content, relevance and coherence of the response. In particular, the meaning of the spoken response is evaluated, making it possible for these models to assess whether or not what was said deserves a high score.

Scoring writing responses with Intelligent Essay Assessor? (IEA)

The written portion of PTE Academic is scored using the Intelligent Essay Assessor? (IEA), an automated scoring tool powered by 色猫AV’s state-of-the-art Knowledge Analysis Technologies? (KAT) engine. Based on more than 20 years of research and development, the KAT engine automatically evaluates the meaning of text, such as an essay written by a student in response to a particular prompt. The KAT engine evaluates writing as accurately as skilled human raters using a proprietary application of the mathematical approach known as Latent Semantic Analysis (LSA). LSA evaluates the meaning of language by analyzing large bodies of relevant text and their meanings. Therefore, using LSA, the KAT engine can understand the meaning of text much like a human.

What aspects of English does PTE Academic assess?

Written scoring

Spoken scoring
  • Word choice
  • Grammar and mechanics
  • Progression of ideas
  • Organization
  • Style, tone
  • Paragraph structure
  • Development, coherence
  • Point of view
  • Task completion
  • Sentence mastery
  • Content
  • Vocabulary
  • Accuracy
  • Pronunciation
  • Intonation
  • Fluency
  • Expressiveness
  • Pragmatics

More blogs from 色猫AV

  • two colleagues working outdoors talking to eachother over a laptop and paperwork

    The role of English proficiency in a global organization

    By Samantha Ball
    Reading time: 4 minutes

    To compete in the modern business space, many organizations are shifting to a more global way of working.

    The COVID-19 pandemic accelerated digital transformation and globalization across industries but, as many businesses look now to sustaining future growth, the question is: what powers a successful, global organization?

    Why English proficiency is key to unlocking international success

    According to 色猫AV’s 2022 Power Skills Survey, one of the biggest factors in effective workplace globalization is the use of English as a common language. The report states “Adopting a common language and ensuring that the entire workforce can access it has become more important than ever before.”

    It points to statistics from rising economies in Southeast Asia, showcasing the strength of English use and their respective booming industries:

    • In Indonesia, Southeast Asia’s largest economy in terms of GDP, the service sector employs nearly half of the country’s local workforce.
    • In Thailand, nearly 60% of the country’s GDP is generated by tourism and export-oriented production. Vietnam, one of APAC’s fastest-growing markets and a rising manufacturing powerhouse, counts the US among its top trading partners.
    • The Philippines’ business process outsourcing (BPO) sector, a key pillar of the country’s economy, employs 1.3 million people.

    The power of effective communication

    色猫AV’s 2022 Power Skills Survey is a drop in the ocean of reports hailing effective communication as one of the top factors to both personal and business success. Good communicators have been linked to higher performance in school, at work and in their personal lives. It is no surprise then that good communicators are highly sought after by talent recruiters.

    With many businesses moving to hiring outside of traditional location boundaries, more languages are inevitable in the workplace. So, can good communication transcend language barriers?

    As the statistics gathered by the Power Skills Survey show, there is evidence that sharing one common language, and therefore enabling more effective communication, can improve business opportunities, workforce growth and ultimately the economy. In the survey report, Norlida Azmi, Group Chief People Officer at Axiata, said:

    “Communication is a critical skill that sets the foundation for not only leadership competencies, such as people management and coalition building, but also for wider and effective stakeholder management in the ecosystem – communicating with our shareholders, customers and the communities that we operate in.”

    English: the official language of business

    While communication for globalization can look different for different markets, the English language is recognized as the prevailing choice for multinational companies. The Harvard Business Review says:

    “The need to tightly coordinate tasks and work with customers and partners worldwide has accelerated the move toward English as the official language of business.”

    So, with businesses now fighting to grow in a global marketplace, unlocking the power of the English language is crucial to success.

    Assessing English proficiency within your organization

    The survey shows that many organizations have work to do in this area, even if they view strong English proficiency as extremely important to their roles. It details the current state of assessing candidates’ English language proficiency and the percentage of organizations that have strategies in place to upskill their employees English proficiency. Both leave room for improvement.

    According to the survey, the number one way talent acquisition managers test English language skills is at the interview (58%), although 37% aren't measuring English language skills at all. This can be problematic later down the line, as not all four language skills (reading, writing, speaking and listening) will be accurately represented in an interview.

    Meanwhile, 27% of businesses use computer-based tests and 14% use paper-based tests to assess English language proficiency. By getting standardized test results, talent managers can more accurately assess if candidates’ English proficiency is suitable for the role, making for more confident and successful hires.

    Strategies for English upskilling

    Talent managers can also pass on these insights to learning and development managers to help them implement effective English language upskilling programs. While this investment can empower employees to perform better in their new role, it may also improve employee retention and even lead to succession and impact business growth.

    ?Whether organizations choose to hire new talent with higher levels of English proficiency or invest in learning and development programs to build their employees’ language skills, there are tools available to help.

    色猫AV Language Solutions for Work enables talent and learning and development managers to fast-track and simplify their recruitment and training processes, giving you total confidence that your candidates and employees have the English language skills to drive your business forward. Find the best language assessment and learning solutions for your organization here.

    Drive your business forward with language training

    If you liked this blog post and want to learn more, download our PDF report here.

    Find out more about how language training and assessment can drive your business forward by checking out our resources for HR professionals, including articles, whitepapers and research.

  • Business woman presenting and discussing with two people in front of her

    How to recruit more effectively: leveraging language assessment AI

    By Samantha Ball
    Reading time: 3 minutes

    There are??why recruiters use artificial intelligence (AI) tools in their hiring process: to save time, provide valuable insights and to make their jobs easier.

    The recruitment process is vital to any thriving business, but it can be time-consuming and labor intensive. Sourcing candidates, screening resumes, conducting interviews and making hiring decisions can be stop-start, with some time-wasting tasks and reprocessing. However, as technology continues to evolve, the role of AI in recruitment is becoming increasingly important.

    With the help of AI, Talent Acquisition leads can improve their recruiting processes, save time and find better-quality candidates. believe AI greatly enhances talent acquisition and retention.

  • Two business people both looking at a computer screen, gesturing and smiling.

    Why your performance management strategy needs language learning

    By Samantha Ball

    The modern workplace is constantly evolving and undergoing a paradigm shift in performance management. The focus is now on fostering a culture of continuous learning and development, rather than merely holding employees accountable. It is particularly important to prioritize this shift when it comes to enhancing language and communication skills within your team.

    It is important to connect performance management with language learning goals, as English skills are highly valued by both employees and businesses.