Virtually Transparent Epidermal Imagery
Abstract
In the process of developing a cyber-physical system capable of displaying the in vivo surgical area directly onto patients' skin, an important research challenge emerged. To generate virtual views from an arbitrary angle, 3D information of internal organ surfaces is crucial. The 3D reconstruction of internal organ surfaces for minimally invasive surgery (MIS) with stereo cameras is usually very difficult due to the challenges in correspondence matching, since there is very limited texture but significant specular reflection on organ surfaces. We have developed a new approach to solve the problem by introducing weakly structured light actively casting surgical tool shadows on organ surfaces.
The contribution is two-fold: first, we propose a robust approach to extract shadow edges from a sequence of shadowed images; second, we developed a novel field surface interpolation (FSI) approach to obtain an accurate and dense disparity map. Our approach does not rely on texture information and is able to reconstruct accurate 3D information by exploiting shadows from surgical tools. One advantage is that the point correspondences are directly calculated and no explicit stereo matching is required, which ensures the efficiency of the method. Another advantage is the minimum hardware requirement because only stereo cameras and a separated single-point light source are required. We evaluated the proposed approach using both phantom models and ex vivo images. Based on the experimental results, we achieved the precision of the recovered 3D surfaces within 0.7mm for phantom models and 1.2mm for ex vivo images. The comparison of disparity maps indicates that with the addition of shadows, the proposed method significantly outperforms the state-of-the-art stereo algorithms for MIS. Our method is not only useful for this project, but also valuable for other image-based medical Interventions.
In addition, to transmit high-definition video from multiple wireless cameras inside human abdomen to a receiver outside of the body, we have designed a potential wireless communication scheme. For our system, the typical optimized metrics in communication schemes, such as power and data rate, are far less important than latency and hardware footprint that absolutely preclude their use if not satisfied. We propose the use of Frequency-Modulated Voltage-Division Multiplexing (FM-VDM) where sensor data is kept analog and transmitted via “voltage- multiplexed” signals that are also frequency-modulated. In this manner the overhead required for digitalization and compression of high-definition (HD) video is removed allowing for a wireless link, which may require higher power to overcome noise, but is possible to devise with a tiny footprint and zero-latency video.
For education purpose, we have developed and implemented a low cost Spatial Augmented Game Environment (SAGE) platform based on our main research techniques. Our current system can superimpose full internal organs, muscles, and skeletons on users. The heart and lung are animated with beating and breathing motion respectively. We also animated lungs with illness and the animation is triggered by a smoking gesture. It is widely believed that increased engagement and spatial interaction may provide benefit to learning. To explore this notion, we have tested an experimental Science, Technology, Engineering, and Mathematics (STEM) educational game entitled ‘Augmented Anatomy’ designed for our proposed platform on a student population. The results indicate that: a) learning of anatomy on-self does reject the null hypothesis and appears correlated with increased engagement and b) a SAGE can be effective at teaching short and long term identification of anatomical structures.
Award ID: 1035594
Submitted by Adam Anderson
on
Abstract
In the process of developing a cyber-physical system capable of displaying the in vivo surgical area directly onto patients' skin, an important research challenge emerged. To generate virtual views from an arbitrary angle, 3D information of internal organ surfaces is crucial. The 3D reconstruction of internal organ surfaces for minimally invasive surgery (MIS) with stereo cameras is usually very difficult due to the challenges in correspondence matching, since there is very limited texture but significant specular reflection on organ surfaces. We have developed a new approach to solve the problem by introducing weakly structured light actively casting surgical tool shadows on organ surfaces.
The contribution is two-fold: first, we propose a robust approach to extract shadow edges from a sequence of shadowed images; second, we developed a novel field surface interpolation (FSI) approach to obtain an accurate and dense disparity map. Our approach does not rely on texture information and is able to reconstruct accurate 3D information by exploiting shadows from surgical tools. One advantage is that the point correspondences are directly calculated and no explicit stereo matching is required, which ensures the efficiency of the method. Another advantage is the minimum hardware requirement because only stereo cameras and a separated single-point light source are required. We evaluated the proposed approach using both phantom models and ex vivo images. Based on the experimental results, we achieved the precision of the recovered 3D surfaces within 0.7mm for phantom models and 1.2mm for ex vivo images. The comparison of disparity maps indicates that with the addition of shadows, the proposed method significantly outperforms the state-of-the-art stereo algorithms for MIS. Our method is not only useful for this project, but also valuable for other image-based medical Interventions.
In addition, to transmit high-definition video from multiple wireless cameras inside human abdomen to a receiver outside of the body, we have designed a potential wireless communication scheme. For our system, the typical optimized metrics in communication schemes, such as power and data rate, are far less important than latency and hardware footprint that absolutely preclude their use if not satisfied. We propose the use of Frequency-Modulated Voltage-Division Multiplexing (FM-VDM) where sensor data is kept analog and transmitted via “voltage- multiplexed” signals that are also frequency-modulated. In this manner the overhead required for digitalization and compression of high-definition (HD) video is removed allowing for a wireless link, which may require higher power to overcome noise, but is possible to devise with a tiny footprint and zero-latency video.
For education purpose, we have developed and implemented a low cost Spatial Augmented Game Environment (SAGE) platform based on our main research techniques. Our current system can superimpose full internal organs, muscles, and skeletons on users. The heart and lung are animated with beating and breathing motion respectively. We also animated lungs with illness and the animation is triggered by a smoking gesture. It is widely believed that increased engagement and spatial interaction may provide benefit to learning. To explore this notion, we have tested an experimental Science, Technology, Engineering, and Mathematics (STEM) educational game entitled ‘Augmented Anatomy’ designed for our proposed platform on a student population. The results indicate that: a) learning of anatomy on-self does reject the null hypothesis and appears correlated with increased engagement and b) a SAGE can be effective at teaching short and long term identification of anatomical structures.
Award ID: 1035594