Two scores people wave at you about AI

Read anything about how AI gets tested and sooner or later somebody waves a number at you. Two of those numbers have names that look like typos, BLEU and recall@k.

You’ve already got the gist from the line up top, so you could stop reading here and be fine.

BLEU say “blue”

Remember the teacher who only gave marks if you used the exact words in the book? BLEU is that teacher.

KeyIamverytired YouIamexhausted matchesmatchesnot in the key

Low score. And you meant the same thing.

• matches the key
• not in the key

You said the same thing as the key and got punished for it. That’s how BLEU grades a computer’s translation. It lines the computer’s sentence up against one a person wrote and counts the matching words. Lots of matches, good score.

People still use it because it’s cheap and it’s instant. You can grade ten thousand sentences over lunch. But it’s a word counter. Whether the sentence means the right thing, it couldn’t tell you.

Recall@k say “recall at kay”

This one is about searching. You type “flight” into your email because you need four things for a trip. You’re tired, so you only look at the first five results.

You search for “flight”

1. 1Flight confirmationfound
2. 2Shoe sale ends Friday
3. 3Hotel bookingfound
4. 4Old receipt
5. 5Rental carfound

you stopped looking here (k = 5)

1. Seat change noticenever shown

3 found out of 4 = 75%

• what you needed
• found
• where you stopped
• missed

Three of your four showed up and the seat change never did, so recall is 75%. The k is the 5. It’s how far down the list you were willing to look. Somebody who says “recall at 10” looked at ten.

Notice it doesn’t mind the shoe sale sitting at number 2. It only counts what you wanted and whether you got it. (Junk has its own score, called precision, for another day.)

So when somebody waves one at you, it’s fair to ask whether real people tried it too, and whether they were happy with what they got.