Explaining computerized English testing in plain English

ɫèAV Languages
a pair of hands typing at a laptop

Research has shown that automated scoring can give more reliable and objective results than human examiners when evaluating a person’s mastery of English. This is because an automated scoring system is impartial, unlike humans, who can be influenced by irrelevant factors such as a test taker’s appearance or body language. Additionally, automated scoring treats regional accents equally, unlike human examiners who may favor accents they are more familiar with. Automated scoring also allows individual features of a spoken or written test question response to be analyzed independent of one another, so that a weakness in one area of language does not affect the scoring of other areas.

was created in response to the demand for a more accurate, objective, secure and relevant test of English. Our automated scoring system is a central feature of the test, and vital to ensuring the delivery of accurate, objective and relevant results – no matter who the test-taker is or where the test is taken.

Development and validation of the scoring system to ensure accuracy

PTE Academic’s automated scoring system was developed after extensive research and field testing. A prototype test was developed and administered to a sample of more than 10,000 test takers from 158 different countries, speaking 126 different native languages. This data was collected and used to train the automated scoring engines for both the written and spoken PTE Academic items.

To do this, multiple trained human markers assess each answer. Those results are used as the training material for machine learning algorithms, similar to those used by systems like Google Search or Apple’s Siri. The model makes initial guesses as to the scores each response should get, then consults the actual scores to see well how it did, adjusts itself in a few directions, then goes through the training set over and over again, adjusting and improving until it arrives at a maximally correct solution – a solution that ideally gets very close to predicting the set of human ratings.

Once trained up and performing at a high level, this model is used as a marking algorithm, able to score new responses just like human markers would. Correlations between scores given by this system and trained human markers are quite high. The standard error of measurement between ɫèAV’s system and a human rater is less than that between one human rater and another – in other words, the machine scores are more accurate than those given by a pair of human raters, because much of the bias and unreliability has been squeezed out of them. In general, you can think of a machine scoring system as one that takes the best stuff out of human ratings, then acts like an idealized human marker.

ɫèAV conducts scoring validation studies to ensure that the machine scores are consistently comparable to ratings given by skilled human raters. Here, a new set of test-taker responses (never seen by the machine) are scored by both human raters and by the automated scoring system. Research has demonstrated that the automated scoring technology underlying PTE Academic produces scores comparable to those obtained from careful human experts. This means that the automated system “acts” like a human rater when assessing test takers’ language skills, but does so with a machine's precision, consistency and objectivity.

Scoring speaking responses with ɫèAV’s Ordinate technology

The spoken portion of PTE Academic is automatically scored using ɫèAV’s Ordinate technology. Ordinate technology results from years of research in speech recognition, statistical modeling, linguistics and testing theory. The technology uses a proprietary speech processing system that is specifically designed to analyze and automatically score speech from fluent and second-language English speakers. The Ordinate scoring system collects hundreds of pieces of information from the test takers’ spoken responses in addition to just the words, such as pace, timing and rhythm, as well as the power of their voice, emphasis, intonation and accuracy of pronunciation. It is trained to recognize even somewhat mispronounced words, and quickly evaluates the content, relevance and coherence of the response. In particular, the meaning of the spoken response is evaluated, making it possible for these models to assess whether or not what was said deserves a high score.

Scoring writing responses with Intelligent Essay Assessor™ (IEA)

The written portion of PTE Academic is scored using the Intelligent Essay Assessor™ (IEA), an automated scoring tool powered by ɫèAV’s state-of-the-art Knowledge Analysis Technologies™ (KAT) engine. Based on more than 20 years of research and development, the KAT engine automatically evaluates the meaning of text, such as an essay written by a student in response to a particular prompt. The KAT engine evaluates writing as accurately as skilled human raters using a proprietary application of the mathematical approach known as Latent Semantic Analysis (LSA). LSA evaluates the meaning of language by analyzing large bodies of relevant text and their meanings. Therefore, using LSA, the KAT engine can understand the meaning of text much like a human.

What aspects of English does PTE Academic assess?

Written scoring

Spoken scoring
  • Word choice
  • Grammar and mechanics
  • Progression of ideas
  • Organization
  • Style, tone
  • Paragraph structure
  • Development, coherence
  • Point of view
  • Task completion
  • Sentence mastery
  • Content
  • Vocabulary
  • Accuracy
  • Pronunciation
  • Intonation
  • Fluency
  • Expressiveness
  • Pragmatics

More blogs from ɫèAV

  • Children sat at desks in a classroom, with one in the middle smiling with her hand up

    Teaching young learners: all just fun and games?

    By Jill Leighton
    Reading time: 5 minutes

    We want lots of fun in our lessons, with a good balance of activities and games with strong educational value. But in my opinion, today’s young learners need more than just fun and games to stay motivated. Fun and games can quickly become meaningless and boring for some students.

    How do we create fun and engaging lessons for young learners aged 6-14?

    • Have a strong relationship with your students. Know about their natural talents, their weaker points and where they are in their development.
    • Encourage them to be active learners and engage them in learning that is meaningful to all of them. E.g. through project work.
    • Provide opportunities for students to use technology and develop success skills. E.g. collaboration, communication, critical thinking, creativity and self-learning.

    Most teachers agree that we have to assume the responsibility of preparing students for life as well as helping them to acquire a language – and this is what I aim to demonstrate in this blog. Using , I want to demonstrate how you can learn more about your students and care for individual needs in one classroom.

    The students will also learn about themselves and their classmates. They will acquire vocabulary and language (at their level) in a dynamic way and create a project to explore their interests and demonstrate their talents. They will use technology and develop their success skills. Assessment will be fun and engaging. I hope you will be able to adapt the ideas and put them into practice with children aged 6-14.

    Howard Gardner’s Model of Multiple Intelligences

    prompts us to ask: How is this child intelligent? He identified eight different types of intelligence which guide the way students learn:

    • Verbal-Linguistic Intelligence (Word Smart)
    • Logical-Mathematical Intelligence (Maths Smart)
    • Spatial Intelligence (Picture Smart)
    • Musical Intelligence (Music Smart)
    • Bodily-Kinesthetic Intelligence (Body Smart)
    • Naturalistic-Environmentalist Intelligence (Nature Smart)
    • Interpersonal Intelligence (People Smart)
    • Intrapersonal Intelligence (Self-Smart)
    • (He later went on to identify Existential Intelligence – Life Smart, and Pedagogical Intelligence -Teacher Smart)

    Lesson 1: Outline the student’s learning goals

    Explain to the students that they are going to learn about Howard Gardner’s Model of Multiple Intelligences and that you want them to achieve learning goals on completion of the project. Write the goals on a poster and discuss them with your students. Take a digital photo of the poster to use during the assessment.

    Student learning goals – you will be able to:

    • Identify different intelligences and what they mean. Name eight different intelligences you have
    • Communicate ideas clearly through a project to demonstrate your talents and interests
    • Use your success skills more effectively.

    Tip: Video parts of lesson 1, especially discussion of the learning goals, to use as part-assessment and reflection on completion of the project.

    What do you know about the Multiple Intelligences? How are you smart?

    • Ask children what they know about multiple intelligences. Teach key language and vocabulary as necessary, depending on level.
    • Ask, “What does it mean be intelligent?” (They will probably say, good grades, be good at maths, English, reading, writing, computers...)
    • Tell them being intelligent or smart (get the younger children to repeat the word smart several times) isn’t only about getting good grades. Ask them to think of more ways to be smart.
    • Elicit the eight ways to be smart according to Gardner. (You may need to mime). E.g. mime playing the piano or clap a rhythm to elicit Music Smart.
    • Discuss what the different intelligences mean. (E.g. Word Smart (Linguistic Intelligence): you like reading, writing or speaking, you are probably good at languages…)
    • Teach career vocabulary associated with the different intelligences. (E.g. Word Smart: journalist, teacher, lawyer, editor, TV announcer, web editor.)

    Lesson 2: Find out your smarts quiz

    Tell students they are going to do a quiz to discover how they are smart. Model each stage of the activity and do the quiz with them to find out about your own strengths. Give each student a piece of paper.

    • Take your paper and show the students how to fold it into eight sections, then unfold it and draw lines along the folds to make a grid.
    • Write the different smarts in each section. (Use small handwriting to leave room to illustrate each smart with a picture).
    • Give a picture dictation to illustrate each smart. Here are some examples:

    Word Smart:Draw a dictionary and children reading, writing and speaking.

    Logic/Number Smart: Draw sums on a computer, a scientist with test tube.

    Music Smart: Draw children singing and playing musical instruments.

    Body Smart: Draw children playing a sport, dancing or cooking.

    Nature Smart:Draw trees, animals, insects, child watering a plant.

    Spatial/Picture Smart: Draw children drawing, painting or taking photos and a pilot in a plane.

    People Smart:Draw a child helping or leading a group or a group of children holding hands.

    Self-Smart: Draw children keeping a journal, researching on a computer, or meditating.

    Encourage students to order their smarts from 1-8. For example, if you love music, write number 1 in the Music Smart section and continue to 8 in order of preference. (You may wish to model this first and order your smarts from 1-8 and then encourage the children to order their smarts.)

    After the quiz

    • Encourage students to compare and discuss their results. Collect the papers and make notes about each student’s results. This will help you reach all your students when planning activities.
    • Explain that we have all the intelligences in different degrees and that all of the intelligences are equal (no intelligence is better than another). Also point out that it is important to know our strengths in order to help in all subjects. (E.g. a music smart student who finds maths challenging may want to sing multiplication tables). Remind students that we usually use several intelligences to do something and we can explore and develop all our smarts.

    Tip: Video lesson 2 to use as part of assessment.

    Lesson 3: Beginning the project

    Encourage the students to create a project for enjoyment. Explain that you want them to collaborate in groups that share the same smarts and interests and using their creativity you want them come up with an interesting topic to explore. (Remind students to concentrate on developing their success skills when they are working with others and mention that you will also be monitoring this as part of the assessment).

    Organize the students into groups of no more than five students in each. Give students time to brainstorm in their groups and come up with the best topic for the project, using their critical thinking skills. (E.g. Picture Smart students may decide to create a project about a famous artist, such as Salvador Dali).

    You may wish to give each student a (Know, Want-to-know, and Learned) to complete during the project process, asking questions such as: What do you know? What do you want to know? What have you learned?

    Lesson 4: Planning the project

    Inform students about the timing of the project. Encourage each group to make a project mind map, which will encourage them to be more creative and organized.

    Remind all the groups to keep the following questions in mind:

    • What are you going to do/make? (Presentation, PowerPoint, website, video, posters.)
    • How will you research it? (Internet, tech tools, library.)
    • How will you delegate responsibilities?
    • How will you check that your audience has understood the message of your project? What questions will you ask?

    Decide on how many lessons are needed to prepare the project and how much will be done in school or at home, depending on the age groups and timing.

    Encourage the students to share their work with the rest of the class (or in assembly.)

    Tip: Video the different groups sharing their work.

    Can assessment also be fun and engaging?

    Yes, it can; here are some tips and suggestions.

    • Show the videos you have recorded and ask the students to compare and contrast their knowledge in Lesson 1 and how it developed over the lessons. Encourage them to observe and comment on their success skills.
    • Display the photo of the Student Learning Goals poster from Lesson 1. Get the students to self-assess and decide whether they have achieved the goals that were set in Lesson 1. (You may wish to give the students three small pieces of colored paper: red to represent I understand quite well, orange to represent I understand well and green to represent I understand very well). Ask: Can you identify the different intelligences and say what they mean? Encourage the students to hold up a colored piece of paper according to their understanding. (Make a mental note of all red pieces of paper to be ready to give extra help to those students). Check understanding by getting several students to answer the question.
    • Get the children to reflect on the learning experience. What have they learned about the different intelligences? How can they develop weaker points using their strengths to help them? Can they use all eight intelligences inside and outside school? Did they manage to get along well with their classmates? Did they communicate the message of their project so that the audience understood?
    • Give individual feedback to each student. E.g. congratulate them on their attitude and effort or identify areas for improvement: “You managed to use vocabulary and language effectively when you shared your project, we understood your message perfectly.” Or “You need to work on being more collaborative.” “You weren’t on task during the project.” “How do you think you can improve that?”
    • Ask students to give you feedback on the activities they enjoyed. Get them to draw happy and sad face cards. Go through all the activities and get the students to show a happy or sad face according to whether or not they liked the activity. E.g. say “Did you like the ‘Find out your smarts’ quiz?” and ask them to hold up the happy or sad face depending on whether they liked the activity or not.

    Create fun lessons to engage all your students keeping this model in mind: traditional activities such as short fun activities and games + Howard Gardner’s model of Multiple Intelligences + PBL (Project Based Learning) + success skills + meaningful assessment. Enjoy the results with your students.

    How the Global Scale of English can help

    TheGlobal Scale of English (GSE) Learning Objectives for provides ready-made learning objectives that can help with planning curriculums and lesson and benchmarking learners’ progress. They are great for young learners because they describe language functions in a granular way, enabling educators to give their learners credit for small achievements. They also clearly show the language functions to target next in order to take learners to the next level.