«A Fast and Flexible Computer Vision System for Implanted Visual Prostheses Wai Ho Li Monash Vision Group, Monash University, Australia Abstract. ...»
A Fast and Flexible Computer Vision System for
Implanted Visual Prostheses
Wai Ho Li
Monash Vision Group, Monash University, Australia
Abstract. Implanted visual prostheses generate visual percepts by electrically stimulating the human visual pathway using an array of electrodes. The resulting bionic vision consists of a spatial-temporal pattern
of bright dots called phosphenes. This patient-speciﬁc phosphene pattern
has low resolution, limited dynamic range and is spatially irregular. This paper presents a computer vision system designed to deal with these limitations, especially spatial irregularity. The system uses a new mapping called the Camera Map to decouple the ﬂexible spatial layout of image processing from the inﬂexible layout of phosphenes experienced by a patient. Detailed simulations of a cortical prosthesis currently in preclinical testing were performed to create phosphene patterns for testing.
The system was tested on a wearable prototype of the cortical prosthesis.
Despite having limited computational resources, the system operated in real time, taking only a few milliseconds to perform image processing and visualisations of simulated prosthetic vision.
Keywords: Visual Prosthesis, Bionic Eye, Cortical implant, Simulated Prosthetic Vision, Wearable computer vision, Integral Images, Irregular, Camera Maps, Real Time, Phosphene Maps, Image processing 1 Introduction According to the World Health Organization, visual impairment and blindness aﬀect nearly 300 million people worldwide1. Some causes of vision loss, such as cataracts, can already be treated using existing medical technology. Implanted Visual Prostheses (IVP) attempt to address currently incurable diseases, such as Retinitis Pigmentosa (RP), by electrically stimulating the still-healthy parts of a patient’s visual pathway to produce prosthetic vision.
Prosthetic vision has many limitations, which are further detailed in Section 2. These limitations severely restrict the bandwidth of visual information that can be provided to a patient. Computer Vision provides a promising way to improve the usefulness of prosthetic vision despite its limitations. This paper presents a computer vision system for implanted visual prostheses. The system can be ﬂexibly tailored in a patient-speciﬁc manner and operates in real time on a computationally limited wearable prototype. The research contributions, design and testing of the system are detailed in Section 3.
http://www.who.int/mediacentre/factsheets/fs282/en/ 2 ECCV-14 submission ID W22-54 Ever since 1755, when LeRoy discharged a Leyden Jar to cause a blind patient to see “ﬂames passing rapidly downward” , numerous experiments have conﬁrmed that electrical stimulation of the human visual pathway can result in visual percepts. Modern implanted visual prosthesis (IVP) operate using the same fundamental principle. Controlled electrical stimulation is applied using small implanted electrodes to produce a bright visual percept called a phosphene.
By apply temporally varying stimuli using an array of electrodes, the patient sees spatial-temporal patterns of phosphenes similar to a low resolution dot display.
In the late 1960’s, Brindley and Lewin  developed the ﬁrst IVP. The system used an array of electrodes on the visual cortex to elicit multiple phosphenes at diﬀerent locations of a patient’s visual ﬁeld. However, the IVP was only suitable for laboratory use as the stimulation electronics were not portable. The IVP also did not include a portable camera.
From the 1970’s to the early 2000’s, Dobelle developed several IVP devices that used implanted cortical electrode arrays, including systems that generate electrical stimuli based on imagery captured with a headworn camera . Despite a range of problems including the heaviness of the portable electronics and the use of wired transcranial connections, a patient’s biography suggests that the device did provide useful vision .
Recent research and development have focused on IVP that electrically stimulate either the retina or the visual cortex2. The reason for the focus on retinal and cortical stimulation is that electrical stimulation at these two anatomical locations can give reliable spatial patterns of phosphenes. Retinal prostheses, such as the Argus II device from Second Sight, have already been implanted into several tens of human patients in clinical trials . Cortical implants, such as the Monash Vision Group’s Gennaris device3, are still in the preclinical phase.
However, cortical implants may be able to treat additional causes of blindness as the cortex is further downstream along the visual pathway. The cortex also has a larger surface area than the retina, which may allow vision with higher spatial resolution.
For a survey of IVP research and development, including many concepts in this paper, please refer to the extensive book edited by Dagnelie .
2 Limitations of Implanted Visual Prostheses At a fundamental level, implanted visual prostheses operate by converting imagery from a headworn camera into spatial-temporal patterns of electrical stimulation applied to a patient’s visual pathway. This is true for both cortical and retinal prostheses. The conversion process is usually performed on portable computational hardware, which is externally worn by the patient.
Figure 1 is a system overview of Monash Vision Group’s cortical visual prosthesis , which contains stereotypical sub-systems shared by many other prostheses. Images are captured by a headworn camera and sent to a portable comhttp://www.eye-tuebingen.de/zrenner/retimplantlist/ http://www.monash.edu.au/bioniceye/resources.html (Annual report 2013) ECCV-14 submission ID W22-54 3 puter, the Pocket Processor. In real time, the pocket processor converts camera images into spatial-temporal patterns of electrical stimulation, which are conveyed over a wireless link. The implanted electrodes receives electrical power and signal from the wireless coil, which it uses to apply electrical stimulation to the visual cortex. A conceptual walkthrough of how the MVG device operates is available online: http://youtu.be/v9Ip8j3eca8.
Fig. 1: Overview of the Monash Vision Group (MVG) Cortical Visual Prosthesis.
2.1 Limited Spatial and Intensity Resolutions The conversion from headworn sensor imagery to electrical stimuli is an ongoing research problem. While state-of-the-art stimulation regimes are able to reliably elicit phosphenes (bright visual percepts), the elicited phosphenes have poor dynamic range and can only be packed at low spatial resolutions. Figure 2 illustrates this using Simulated Prosthetic Vision (SPV), a technique pioneered in the early 1990’s to simulate what an implanted patient may see . The input image is converted into prosthetic vision using an adaptive thresholding approach  where a corresponding phosphene is enabled for bright regions of the input image. The SPV assumes the ability to generate 625 binary phosphenes, which is similar to the expected capabilities of the Monash Vision Group prosthesis .
The SPV image in Figure 2b clearly illustrates the severe information loss due to the limited spatial and intensity resolution of prosthetic vision. As the number of phosphenes generally corresponds to the number of implanted electrodes4, the spatial resolution of prosthetic vision is limited by the factors such as the spread coordinated activation of many electrodes may increase future phosphenes counts 4 ECCV-14 submission ID W22-54
Fig. 2: Simulated Prosthetic Vision (SPV) from an implanted visual prostheses.
of electrical charge, surgical safety of implantation and electrode fabrication technology. Improvement in these areas are slow as they often require lengthy preclinical and clinical trials.
Clinical studies of retinal prostheses  suggest that multiple levels of phosphene brightness can be achieved but brightness ratings are likely to vary substantially across sessions and across subjects. There is also evidence that phosphenes of multiple levels of intensity can be produced by varying stimulation currents , but changes in phosphene brightness may be coupled with changes in phosphene shape and size. There is little evidence that phosphene brightness can be varied consistently with a cortical prosthesis. As such, the work presented below assumes the worst case of binary phosphenes.
Arguably, Dobelle was the ﬁrst to consider the use of computer vision to improve the usefulness of prosthetic vision . More recently, simple IVP computer vision algorithms were developed to run on wearable devices with embedded processors [29, 23]. More sophisticated IVP vision algorithms have also been investigated using less portable computers. Transformative Reality  uses multiple sensing modalities to better render a pattern of phosphenes representing models of the world. The substantial body of work on Vision Processing for prosthetic vision  applies computer vision algorithms to improve simulated multi-intensity phosphene patterns for reading text, navigation and other tasks.
2.2 Irregular Phosphene Maps
A patient’s Phosphene Map contains all the phosphenes that can be elicited by electrical stimulation. Older IVP research, including work on image processing and computer vision, generally assumes regular phosphene maps similar to the map shown in Figure 3a. However, there is strong clinical and biological evidence to suggest that phosphene maps are irregular and patient-speciﬁc [3, 19, 27]. An example of an irregular phosphene map is shown in Figure 3b.
Apart from irregular locations and sizes, there is also evidence that phosphenes can exhibit irregular shapes. Studies from Bionic Vision Australia5 and Second http://goo.gl/LwcGwO ECCV-14 submission ID W22-54 5
Fig. 3: Example of Regular and Irregular phosphene maps.
Sight  show that the shape of phosphenes can be anisotropic and the shape of phosphenes may vary depending on electrical stimulation.
The computer vision system presented in this paper has the potential to accommodate all three aspects of phosphene map irregularity: location, size and shape. However, as the MVG device is still in the preclinical stage, the system is only tested on phosphene maps simulated based on electrical, surgical and cortical models. These models only generate irregularities in phosphene locations and sizes. The simulation assumes that phosphenes appear as isotropic Gaussians as recommended by the survey of simulated prosthetic vision by Chen et al .
Details of the simulation are available in Section 3.1.
3 Computer Vision System for IVP Despite the reality that Implanted Visual Prostheses (IVP) produce irregular phosphene maps, very little research has been done to address the problem in full. Research that attempts to deal with irregular phosphene maps generally only do so for near-regular mappings where small spatial shifts in phosphene locations and electrode dropouts are modelled  or only irregular phosphene shapes are considered over a regular grid .
More importantly, many systems do not run in real time on an embedded processor suitable for a wearable medical device. Clinical trials of retinal implants  and cortical implants  suggest that prosthetic vision may have refresh rates as high as 10Hz. Simulated Prosthetic Vision trials show that low refresh rates may reduce task performance . Therefore, a practical IVP requires a fast and ﬂexible computer vision system.
Given the background above, this paper provides the following contributions:
1. Section 3.1 describes a detailed simulation of a cortical IVP device
2. Section 3.2 details a computer vision system that deals with irregular phosphene maps using a second mapping called the Camera Map to provide ﬂexibility.
3. Section 3.3 details a fast image processing method for the vision system.
4. Section 3.4 details a simulated prosthetic vision visualisation that shows the phosphenes seen by a patient in real time.
5. Section 3.5 summarises the real time performance of the system.
6 ECCV-14 submission ID W22-54
3.1 Simulating a Cortical Implanted Visual Prosthesis
Phosphene maps were simulated in order to test the computer vision system.
The simulation is based on the Monash Vision Group (MVG) cortical Implanted Visual Prothesis (IVP), which is currently undergoing preclinical trials. Parameters of the MVG IVP system were obtained from published sources [17, 16].
The main components of the simulation are detailed on the left of Figure 4 (in red). The simulation starts with the deﬁnition of the spatial layout of an implanted electrode array. The array is also known as a tile. The MVG IVP uses multiple identical tiles. A tile contains 43 active electrodes. The MVG Tile Layout is shown at the top left of Figure 4 with blue dots representing electrodes.
Next, the simulation places multiple tiles onto the surface of the visual cortex. Coordinates on the visual cortex are deﬁned on a Cortical Plane, which represents a ﬂattened cortical surface. Tile Locations are deﬁned using 2D aﬃne transforms. This results in a list of Ideal Electrode Locations on the cortical plane.
A surgical scenario proposed by a MVG Neurosurgeon is shown at the middleleft of Figure 4. The four-tile wedge-shaped arrangement avoids the Calcarine Sulcus, which is a large crevice on the visual cortex.
The simulation then applies two sources of irregularities that simulate real world issues: Electrode dropouts and the imprecise placement of electrodes. The Dropout Rate models implanted electrodes that fail to elicit a phosphene when stimulated electrically. For example, a dropout rate of 50% means that half of all implanted electrodes cannot be used to elicit a phosphene. Electrode dropouts have been reported in multiple IVP clinical trials [11, 32], but generally at rates lower than 50%.
Spatial Error models several issues by approximating their combined eﬀect as a normally distributed 2D random oﬀset deﬁned on the cortical plane. For example, electrode deformation during surgical insertion and variations in cortical anatomy are both factors that can be approximated as spatial error. The application of dropouts and spatial error results in Irregular Electrode Locations, an example of which can be seen at the bottom-left of Figure 4.