Pandemic Package

Use Case

We built a system that utilizes a NVIDIA Jetson to visually detect packages and - if existent - damaged areas on the detected packages. In case of a detected damage, a connected phone is notified with a brief description and a picture of the damage.

Notification of damaged parcel

With online trade becoming more and more popular, the number of shipments (in Germany alone) rose to more than 3.5 billion in 2018, expected to reach 4.4 billion in 2023. Handling this immense daily volume requires logistics and distribution centers to consistently increase efficiency, while maintaining a sufficient level of quality. One crucial detrimental factor in this context are damaged or unsealed packages. To enable quality control and maintain customer satisfaction, extensive and expensive manual effort is required to evaluate the intactness of outgoing shipments. Moreover, if not identified in time, damages can cause interruptions and traffic jams within the material flow.

Model Evaluation

Value Proposition

We want to resolve this issue by implementing an automated warning system for logistics and distribution centers that informs the control center, whenever some quality specification of a package is violated. This can be set up and used in receiving areas, before shipping, or everywhere within the supply chain. In this sense, it might also be of interest for package delivery services, such as DHL, Hermes, and DPD, to avoid handing out corrupted parcels. Our computer vision model firstly detects packages and secondly localizes and classifies critical areas of a package. The picture including the critical information is immediately transmitted to the control center (in our case a smartphone) to intervene and resolve the issue in time. This way, a high quality of packages within the flow system can be guaranteed and corrupted packages can be filtered out quickly, all while saving manual effort.

Proof of Concept

For collecting training data and providing a proof of concept, we organized a cooperation with an industry partner who operates a distribution center. Unfortunately, due to Corona Virus restrictions, our visit to the logistics center had to be cancelled

Outlook

Real-Life Insights

We would have loved to apply real-life insights and fine tune for the working environment. Based on domain knowledge from logistics experts, we could have specified critical properties.

Two-Stage Model Architecture

One of our initial ideas was to implement an additional damage detection model that is applied to the identified bounding boxes to factor out the damage detection as a sequential step. In other words, as soon as a package would be identified by our “base model”, the focal pixel coordinates would be rescaled and applied to the original image so that the more complex damage analysis would be based only on the package itself. This way, for inference the model would be able to feed in the maximum of relevant pixels, hopefully improving damage detection. Also, this architecture would implicitly encode that the damages can only occur on packages.

Implementation of Polygon Labels & Instance Segmentation

Our model was initially planned to read in and process polygon annotations for the damages. We assume that accounting for the more precise polygons would significantly increase mAP for the damages. Also, it would be valuable for instance segmentation, which is a possible future enhancement, but out of scope for this project.

Deployment: Integration into a Warehouse Management System (WMS)

Finally, with regards to deployment, our service is perfectly suited to be integrated within a WMS and thus be embedded within a service system.