Amazon taps AI, ML to keep bad apples from its shelves

In an interview, Rajeev Rastogi, vice-president of ML at Amazon India, said the company has developed computer vision programs that recognize defects such as cuts and scratches on tomatoes and onions to figure out when they have gone bad.

The system uses a mix of convolutional neural networks (CNNs) and visual transformer (ViTs) algorithms. CNNs are deep learning algorithms that can take image input and assign importance to various aspects of that image, while ViTs are specialized versions of transformer algorithms, which can weigh the significance of each part of data it gets.

“In our grocery business, produce quality is the single-most important customer input and the number one driver of repeat purchase,” Rastogi said. “Currently, quality is processed manually, which doesn’t really scale. It’s also very error-prone, is costly and doesn’t have high repeatability. So, we developed a computer vision system for grading fresh produce quality by analysing images of produce,” he said.

Amazon uses computer vision techniques to detect cuts, cracks and pressure damages and more. The tech has been deployed for tomatoes and onions now in Amazon’s store, but Rastogi said it is building an AI-enabled machine, Auto Grader, to automatically grade produce moving on a conveyor belt.

“It will improve the quality and consistency of the produce while reducing the grading cost by almost 78% as compared to manual grading,” he said.

“In future, we also want to use infrared sensors to detect attributes like sweetness and ripeness, which we cannot detect in RGB (red, green and blue) images that are captured by your traditional computer vision algorithms,” Rastogi said. He noted that infrared can help avoid “destructive methods” of quality assurance, like how ripeness and sweetness often require a person to actually eat a fruit or vegetable.

“A near-infrared image will be very different for a sweet versus not sweet product. Because the level of sugar will be different, and the near-infrared signature will be different,” he said.

The system is deployed in stores in India, along with some pilot projects in Europe as well. Rastogi admitted that the system is at an early stage and it will be a while before the company could place timelines on exactly when it will be rolled out at scale. However, he said that the system is at above 90% accuracy in terms of precision in finding defects. “It varies from one produce to another. Like we have much better results for tomatoes and we can do it with high precision. Our numbers are a bit lower for onions and we are working on (improving that).”

The difficulty in determining the quality of fresh produce can also differ from one product to the next, he said. “Most defects that we are looking at (right now) can be visually detected by a human. Anything that can be visually detected, I think, can be easily detected by computer vision as well,” he said.

In future, Amazon could use a system like this to classify and categorize products to recognize the kind of product offered by a seller. For instance, if a seller is offering a particular type of tomatoes, the system could categorize what that tomato should look like and figure out in advance whether the right product is being delivered.

The company has also had to tailor the system to Indian produce quality. As the quality of fresh produce in India isn’t the same as counterparts in Western countries, the approach to determining quality has to change as well. “A tomato in India is not the same as a tomato in the US. So, you can’t just say that you will have the same approach to dealing with tomatoes and onions in every country. The nature of the product could be quite different, because of farming practices,” Rastogi said.

In theory, the company could also bring the Auto Grader to the front-end, where customers could just open the Amazon app and detect the quality of a vegetable or fruit by themselves.