How to implement Deep Learning in the food industry?
So I have one idea. I wrote this in my Ph.D. two years ago. But since it’s not put yet in the procedure, I decided to write it here. I hope that someone who has some relations in this industry gives me feedback about this and maybe this idea finds a way to realize it.
Of course that I took a few serious conversations with some people who have been used the standard approach of procedure of color sorting and the ones who have experience in the implementation of Deep Learning alg, but it’s hard to gain the real gamers of the production lines like Key and Raytec Vision (I have some experience with their color sorters machines). And one of my biggest wishes is to visit the Key (for a start) :) Manufactures of color sorting machines are my target group.
So let me start from the beginning. I wrote here in two stories (1, 2) about Machine Vision, color sorting, and some parts of my research. I have used the pictures of raspberries from the color sorting machines which have taken during the process of sorting inspection. Mechanical inspection in industrial plants is one of the key points in the entire processing system. In the segment of the line where the inspection is performed, the so-called color sorting monoblock, consists of the main machine — color sorter, vibrating distributor, and inspection transporter. The color sorter serves one operator who creates recipes for each product (in agreement with the quality controller) what is acceptable and what fruits are not. Non-compliant products are products that do not meet the physical characteristics specified in the class regulations or other materials that can be found during the sorting such as soil, wood, plastic, glass, metal, leaves, parts of animals and insects, etc.
Industrial systems for machine inspection of agricultural products are mostly standardized. Each machine has its own control system and algorithm-based which defines the criteria and selects the product. The main difference is in monitoring systems. Mainly are in use color cameras, infrared cameras, or spectroscopic cameras. Some machines use additional devices such as lasers. The machine can use one or more optical devices. The output accuracy of the color sorter ranges from 92 to 98%. One human operator maintains the machine and creates recipes for each product.
The problem
When creating recipes based on which acceptance or rejection is performed on tested products, the largest role is play the human operator. Potential challenges related to operators are as follows: subjective feeling, insufficient training, frequent replacement of jobs, etc. According to the subjective feeling, the operator checks the output product from time to time and if he notices certain deviations in the form of the passage of a non-compliant product on a part of the line for good the product or if the agreed product is found on a conveyor leading to waste, it is necessary to reprogram the machine, ie change the initial recipe. The color sorting operation in infrequent situations put to shut down, so the machine continues to run for a certain period of time according to an already defined recipe, which can represent a certain loss.
In my research, I used Yolo v3 algorithm for detection, localization, and classification. For now, I will skip the details about training and testing because it’s still part of an unpublished Ph.D. The result was great after several training sessions. The F1 score was between 92–98% where the same results of accuracy also give current sorting machines with standard algorithms. The difference is that with deep learning methods we can achieve almost 100% of accuracy as human experts with enough examples for training. And the biggest achievement will be to increase the objectivity of the whole process.
Why is even 1% enough?
Depending on the processed product, the amount of separated waste on individual machines in the technological line is different. But if we knew now that the amount of processed product per day is between 60 thousand kg to 1 million kilograms, even 1% of lost products present lost money for the company.
My solution
The whole system named CSCS (Centralized System of Color Sorting). Look at the picture below.
Basically, it is an integrated online algorithm update system. The main goal is to reduce the subjectivity of the operator when deciding on the correctness of the product being selected. The presented centralized system is an intelligent solution using modern online technologies together with deep learning algorithms for detection, localization, and classification of objects, in this case, agricultural products. One of the ideas is to implement the Centralized System in a factory that produces color sorters and thus provides absolute online support in the implementation and use of its systems.
How does it work?
The figure above gives a schematic representation of the proposed system consisting of a color sorting machine located in an industrial plant and having the ability to connect to the Internet. That is, the entire connection would be made via the internet connection between the central system and the machine. The central system consists of a hardware part located in the development center of the manufacturer where the labeling data, training, and testing of the algorithm are performed. The obtained weights are sent to the Cloud, which collects the data and sends them in real-time to the color sorter, which from that moment uses a new recipe according to the obtained weights. As an auxiliary option in an industrial plant, there may be a web application for desktop pc and mobile phones that serve operators or quality controllers to check the current recipe or to send additional images to the centralized system for training or class change.
In the meantime, a beta version of such an application has been created.
At any time, the quality controller can take a camera shot of the product on the line in the exact place provided and get information about the product class. The current application contains predefined weight factors trained using the YOLOv3 algorithm and possible classes for products that are sorted on the production line (I created 5 classes). The application is open source and is posted on the GitHub development platform at the following link: https://github.com/ultralytics/yolov3. The web application is written in the Flask web framework based on the Python programming language. The source code of the application and the training configuration can be found on the GitHub repository at the following link: https://github.com/IvaMark/YoloV3-Raspberry.
There are some details about these in the Ph.D. and for now, this is enough information. Who want to see more, let me know by sending a message here or email me at imedojevic@mas.bg.ac.rs or ivanammedojevic@gmail.com.
In addition to weight factors trained on the example of raspberries, image detection can be performed using weight factors trained on the COCO dataset (330 thousand images, 80 classes) created for yolov3, yolov3-spp, yolov3-tiny. It is possible to add a larger number of predefined parameters and thus expand the range of agricultural products that can be tested. The next option of the application is to select the display of predefined classes for certain agricultural products. The advanced functionality of the application is uploading new images that are intended for training. The added image is automatically sent to an email service that is directly connected to the central web server where new images are added to the training set and new training is started. In this way, the existing weights are updated and the application is sent the option to update the new version or set the new version as one of the offered sets of weights. The developed application is a beta version that can be adapted to different agricultural products and the requirements of both processors and manufacturers of color sorters. The application can also be intended for quality controllers or other responsible persons in the factory for further analysis and improvement of production.
What is the main difference from current systems?
The difference would be using deep learning systems that require a minimum role of a qualified person (human operator), data that is regularly updated, an algorithm that is constantly improved and can reach the accuracy of a human expert, remember every case of an undesirable product, data is stored in Cloud and are always available on demand.
The whole idea is to minimize the human subjective influence of product compliance decisions because over time the machine learns through experience. With the development of unsupervised learning algorithms, this possibility is increasing along with the processing of larger amounts of data in real-time. It is necessary to start and create a solution that applies deep learning in such industrial production systems. By applying the proposed idea, the need for technical support directly in the field (eg. industrial plants/lines) is reduced because the machine is programmed online directly by the manufacturer. On the other hand, the machine manufacturer creates its own database on the basis of which it can develop additional applications and services that it can offer in the future and thus constantly achieve a presence and growth in the market. In the very near future, data will be the main currency of exchange.
Competition
Currently, no one uses deep learning algorithms for detection, classification, and localization in color sorters.
What else would it mean for the manufacturer? Users or buyers of the machines would be ‘dependent’ on it. At first glance, the user is absolutely taken care of and everything is available to him at all times, and on the other hand, a centralized system allows the manufacturer to constantly receive data to further develop their own algorithms, create databases and offer constant improvements or opportunities for the user. In the very near future, data will be the main exchange currency. The one who has the most data will always have the advantage.
AT THE END
This is my story and original idea and the main goal of this post is to be free for everyone to read it and send me feedback. This is just a part of the paper and part of the Pitch for the whole model CSCS which I wrote if I had the opportunity someday to present it in front of some development department. But I am happy that I have the opportunity to put this online and maybe I write totally new history of the new direction and approach how new technology like Deep Learning can be used in the future in the agriculture and food industry. I am waiting almost two years to put my research in the ether because some people and some institutions have more power than they should have. So this is new age and ideas can find a different way to be published and to come to the right people and institutions.
‘I don’t care that they stole my idea…I care that they don’t have any of their own.’ — Nikola Tesla
I know that in the near future (maybe I see it tomorrow) DL algorithms will be implemented in color sorters, but I am glad that in some way I am a part of it.
