Explaining computerized English testing in plain English

ɫèAV Languages
a pair of hands typing at a laptop

Research has shown that automated scoring can give more reliable and objective results than human examiners when evaluating a person’s mastery of English. This is because an automated scoring system is impartial, unlike humans, who can be influenced by irrelevant factors such as a test taker’s appearance or body language. Additionally, automated scoring treats regional accents equally, unlike human examiners who may favor accents they are more familiar with. Automated scoring also allows individual features of a spoken or written test question response to be analyzed independent of one another, so that a weakness in one area of language does not affect the scoring of other areas.

was created in response to the demand for a more accurate, objective, secure and relevant test of English. Our automated scoring system is a central feature of the test, and vital to ensuring the delivery of accurate, objective and relevant results – no matter who the test-taker is or where the test is taken.

Development and validation of the scoring system to ensure accuracy

PTE Academic’s automated scoring system was developed after extensive research and field testing. A prototype test was developed and administered to a sample of more than 10,000 test takers from 158 different countries, speaking 126 different native languages. This data was collected and used to train the automated scoring engines for both the written and spoken PTE Academic items.

To do this, multiple trained human markers assess each answer. Those results are used as the training material for machine learning algorithms, similar to those used by systems like Google Search or Apple’s Siri. The model makes initial guesses as to the scores each response should get, then consults the actual scores to see well how it did, adjusts itself in a few directions, then goes through the training set over and over again, adjusting and improving until it arrives at a maximally correct solution – a solution that ideally gets very close to predicting the set of human ratings.

Once trained up and performing at a high level, this model is used as a marking algorithm, able to score new responses just like human markers would. Correlations between scores given by this system and trained human markers are quite high. The standard error of measurement between ɫèAV’s system and a human rater is less than that between one human rater and another – in other words, the machine scores are more accurate than those given by a pair of human raters, because much of the bias and unreliability has been squeezed out of them. In general, you can think of a machine scoring system as one that takes the best stuff out of human ratings, then acts like an idealized human marker.

ɫèAV conducts scoring validation studies to ensure that the machine scores are consistently comparable to ratings given by skilled human raters. Here, a new set of test-taker responses (never seen by the machine) are scored by both human raters and by the automated scoring system. Research has demonstrated that the automated scoring technology underlying PTE Academic produces scores comparable to those obtained from careful human experts. This means that the automated system “acts” like a human rater when assessing test takers’ language skills, but does so with a machine's precision, consistency and objectivity.

Scoring speaking responses with ɫèAV’s Ordinate technology

The spoken portion of PTE Academic is automatically scored using ɫèAV’s Ordinate technology. Ordinate technology results from years of research in speech recognition, statistical modeling, linguistics and testing theory. The technology uses a proprietary speech processing system that is specifically designed to analyze and automatically score speech from fluent and second-language English speakers. The Ordinate scoring system collects hundreds of pieces of information from the test takers’ spoken responses in addition to just the words, such as pace, timing and rhythm, as well as the power of their voice, emphasis, intonation and accuracy of pronunciation. It is trained to recognize even somewhat mispronounced words, and quickly evaluates the content, relevance and coherence of the response. In particular, the meaning of the spoken response is evaluated, making it possible for these models to assess whether or not what was said deserves a high score.

Scoring writing responses with Intelligent Essay Assessor™ (IEA)

The written portion of PTE Academic is scored using the Intelligent Essay Assessor™ (IEA), an automated scoring tool powered by ɫèAV’s state-of-the-art Knowledge Analysis Technologies™ (KAT) engine. Based on more than 20 years of research and development, the KAT engine automatically evaluates the meaning of text, such as an essay written by a student in response to a particular prompt. The KAT engine evaluates writing as accurately as skilled human raters using a proprietary application of the mathematical approach known as Latent Semantic Analysis (LSA). LSA evaluates the meaning of language by analyzing large bodies of relevant text and their meanings. Therefore, using LSA, the KAT engine can understand the meaning of text much like a human.

What aspects of English does PTE Academic assess?

Written scoring

Spoken scoring
  • Word choice
  • Grammar and mechanics
  • Progression of ideas
  • Organization
  • Style, tone
  • Paragraph structure
  • Development, coherence
  • Point of view
  • Task completion
  • Sentence mastery
  • Content
  • Vocabulary
  • Accuracy
  • Pronunciation
  • Intonation
  • Fluency
  • Expressiveness
  • Pragmatics

More blogs from ɫèAV

  • Grammar 101: insider tips and tricks to instantly improve your writing (part 4)

    By
    Reading time: 7 minutes

    Punctuation makes your writing easier to read and understand, but it can be tricky to master. As an editor and proofreader, I often notice people confusing semi-colons and colons, so we'll explore the difference between them. And because both are often used in lists, we'll also look at the humble comma – and its sometimes-controversional cousin, the Oxford comma.

    Semi-colons and colons both connect phrases in a sentence but are used in different situations.

    Understanding colons

    Colons introduce important information and explanations. They're often used before lists as a replacement for phrases like "they are" and "which is":

    • He offered me a choice of drinks: tea, coffee or hot chocolate.
    • I packed the essentials in my bag: water, pens and a magazine.
    • She speaks three languages: English, French and Portuguese.

    You can also think of a colon as a spotlight, with the phrase that comes after the colon explaining or expanding what came before it.

    • In 1903, travel was changed forever by an important event: Orville and Wilbur Wright's first successful flight.
    • He loves visiting the animals at the farm: cows are his favourite.
    • There is one rule I live by: I treat others as I wish to be treated.

    The secrets of semi-colons

    A semi-colon links two ideas that are closely related and that would be two complete sentences if you used a period instead. They give a softer transition than a period would, and they're often used instead of conjunctions like "and", "but" and "because":

    • I love eating pizza; my sister loves eating burgers.
    • I wanted to go for a swim; I couldn't find my goggles.
    • It was the best of times; it was the worst of times.

    Semi-colons also seperate items in long lists to make life easier for the reader and stop a sentence becoming a sea of apostrophes. For example:

    • I've got my shopping list ready: peppers, carrots and oranges from the market; toothpaste, shampoo and pain relief from the drugstore; and a newspaper, snack and drink from the newsstand.

    Standard comma or Oxford comma?

    An Oxford comma goes before "and" or "or" at the end of a list. The first example has an Oxford comma, the second doesn't.

    • Please bring me a sandwich made with cheese, lettuce, and tomato.
    • Please bring me a sandwich made with cheese, lettuce and tomato.

    American English generally favors the Oxford comma, British English typically omits it, unless needed for clarity. Compare:

    • I love my parents, Taylor Swift and Keanu Reeves.
    • I love my parents, Taylor Swift, and Keanu Reeves.

    As with many areas of punctuation, whether you choose to use the Oxford comma is a matter of personal preference. However, the most important thing is to be consistent in your usage.

  • A person in a denim jacket and striped shirt holds glasses and a notebook, standing by a window with bright daylight.

    What happens in the brain when you learn a language?

    By
    Reading time: 7 minutes

    Whether you’re picking up Spanish for travel, Mandarin for business or French just for fun, you’re not only expanding your communication skills, you’re also giving your brain a powerful workout. But what actually happens inside your brain when you learn a language?

    The brain’s language centers

    Your brain is made up of many parts and two areas are significant for language:

    • : Located in the frontal lobe, this region helps you produce speech and form sentences.
    • : Found in the temporal lobe, this area helps you understand spoken and written language.

    When you start learning a new language, these areas get busy. They work together to help you listen, speak, read and write in your new language (Friederici, 2011).

  • What’s it like to teach English in Nepal?

    By
    Reading time: 3 minutes

    Anandi Vara was trained in teaching English in Kathmandu, Nepal before teaching at a monastery in Pokhara. There she taught students ranging from six to 10 years of age, both in groups and individually. Here she reveals the lessons she learned during her time there– including how to avoid being perturbed by a cockroach attack.

    Whatever you do: don’t freak out

    It just makes everything worse. It’s easy for things to get overwhelming – a sense that can be made worse by the feeling of homesickness, especially if it’s your first time living abroad – but thinking rationally, and getting to the source of what’s causing the worry, usually helps. It’s important, as you don't want to share your fear in lessons because you’re the teacher and need to show confidence.

    This was, however, tested to the limit when I had a cockroach dangled in front of my face. It took all my strength to stay calm. I gave an unimpressed look, thereby establishing myself as the figure of authority, which seemed to work.

    Be aware of cultural traditions

    It is important to remember that every country has its own traditions. I was teaching in a monastery, so I made sure to wear respectful clothing, even in the face of soaring temperatures. The more I learned about the Tibetan culture, the more fascinated I became by it. The students taught me how to write my name in Tibetan and the meaning behind it. I learned about Tibetan history and Tibetan culture.

    I also found that the more I showed willingness to learn about the Tibetan culture, the more I bonded with the students, so that when it was time to teach, the students were more cooperative in lessons, engaging and participating more.

    Teaching is two-way learning

    There is so much I learned teaching abroad, both in the classroom and out. Making mistakes as you begin is only natural, but it’s whether you can learn from these mistakes that counts. No two students are ever the same so it’s a constant process of learning as you go. As a result, I learned about the environment I was in – from traditional prayer ceremonies to the Tibetan alphabet– and about myself, notably organizational skills and a renewed curiosity about the English language.

    Be Flexible

    Sometimes it doesn’t matter if you’ve planned your lesson down to the smallest detail – if it doesn’t take, then it doesn’t take. I was only 10 minutes into a lesson once and I could tell that I was beginning to lose the students’ attention. Not only did it show that they were uninterested, it also distracted me from what I was doing. It was at this point that I threw out my existing plan and tried a whole new lesson: I had the students up on their feet and engaging with each other and, although completely improvised, it was very successful.

    Patience is a must

    During my one-on-one mentoring session, my student seemed to have no motivation. He wasn’t learning as well as the other students and had therefore given up. No matter what I tried, he refused to cooperate, but I didn’t let it put me off. I kept trying different methods until finally finding one that he responded to. I made sentence structuring into a game. It wasn’t anything fancy and consisted of scraps of paper with words written on them.

    Although it took a lot of time to find the right angle, it was worth it because he soon realised that although it might take longer for him to pick things up, he would eventually get there and have a greater sense of accomplishment.