Many smartphone applications are designed for screen display on the assumption that they will be operated with the right thumb.As the screen size increases, it may be difficult to operate in other gripping postures.By estimating the gripping posture of the smartphone (which hand to hold and which finger to operate), it is possible to display the screen according to the gripping posture, but the need for an external sensor and the estimation model depend on the smartphone model. There was a problem such as doing.

　Since the screen of the smartphone emits light, if you hold the smartphone in front of your face, the corneal reflex image in the shape of the screen will appear, but the part where you put your finger on the screen will be a shadow, and only that part will be the corneal reflex image. Is missing.Since the way the corneal reflex image is chipped differs depending on the gripping posture, take a facial photograph with the built-in front camera, cut out the corneal reflex image reflected in the pupil from the facial photograph, and classify the corneal reflex image by machine learning. The research group thought that it could be estimated.Since this method uses only the built-in front camera, it is possible to identify the gripping posture of the smartphone alone, and the estimation model does not depend on the smartphone model.

　We verified whether it was possible to identify the gripping posture of 13 experimental collaborators.As a result of creating an estimation model using deep learning, the gripping posture was identified with an accuracy of 85%.In the future, it is expected that the incorporation of this method will help improve the operability of smartphone applications and prevent diseases caused by using smartphones in the same gripping posture for a long time.

Paper information:[In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems] Reflec Touch: Detecting Grasp Posture of Smartphone Using Corneal Reflection Images