Prevention of Operational Errors in Semi-Automatic Riveters by Machine Vision Systems Using Deep Learning

This paper reports the development of an operation support system for production equipment using image processing with deep learning. Semi-automatic riveters are used to attach small parts to skin panels, and they involve manual positioning followed by automated drilling and fastening. The operator watches a monitor showing the processing area, and two types of failure may arise because of human error. First, the operator should locate the correct position on the skin panel by looking at markers painted thereon but may mistakenly cause the equipment to drill at an incorrect position. Second, the operator should prevent the equipment from fastening if they see chips around a hole after drilling but may overlook the chips; chips remaining around a drilled hole may cause the fastener to be inserted into the hole and fastened at an angle, which can result in the whole panel having to be scrapped. To prevent these operational errors that increase production costs by requiring repair work, we have developed an operation support system that processes the monitor images so that the operator can distinguish markers before drilling and detect chips before fastening. Initially, we developed rule-based image processing, but it could not achieve sufficient accuracy because of the complexity of defining rules related to images features. Therefore, we turned instead to image processing based on deep learning, and after efforts to achieve the required accuracy and processing speed, the developed system now outperforms the rule-based system and we have improved the production efficiency of this riveter. Deep learning technology can be used to improve the productivity of a wide range of production equipment, both existing and new.