F21-STAT 940-Proposal
Use this format (Don’t remove Project 0)
Project # 0 Group members:
Abdelkareem, Youssef
Nasr, Islam
Huang, Xuanzhi
Title: Automatic Covid-19 Self-Test Supervision using Deep Learning
Description:
The current health regulations in Canada mandate that all travelers arriving who have to quarantine take a Covid-19 Self-test at home. But the process involves a human agent having a video call with you to walk you through the steps. The idea is to create a deep learning pipeline that takes a real-time video stream of the user and guides him through the process from start to end with no human interference. We would be the first to try to automate such a process using deep learning.
The steps of the test are as follows: - Pickup the swab - Place the swab in our nose up to a particular depth - Rotate in place for a certain period of time - Return back the swab to the required area - https://www.youtube.com/watch?v=jDIUFDMmBDo
Our pipeline will do the following steps: - Use a real-time face detector to detect bounding box around User's face (Such as SSD-MobileNetV2) - Use a real-time landmark detector to accurately detect the location of the nose. - Use a novel model to classify whether the swab is (Inside Left Nose, Inside right Node, Outside Nose) - Once the swab is detected to be inside the nose, the swab will have a marker (such as aruco marker) that can be automatically tracked in real-time using opencv, we will detect the location of the marker relative to the nose location and make sure it's correctly placed and rotated for the determined period. - This whole pipeline will be developed as a state machine, whenever the user violates the rule of a certain step he goes back to the corresponding step and repeats again.
Challenges: - There is no available dataset for training our novel model (to classify whether the swab is inside the nose or not), we will need to build our own dataset using both synthetic and real data. We will build a small dataset (5000-6000 images) and will rely on transfer learning to finetune an existing face attribute (such as moustache and glasses) classification model on our small dataset. - The whole pipeline will need to operate at a minimum of 30 FPS on CPU in order to match the FPS of a real-time video stream.