FREE ELECTRONIC LIBRARY - Dissertations, online materials

Pages:   || 2 | 3 | 4 |

«Zhiwei Zhu a, Qiang Ji b a E-mail:zhuz Telephone: 1-518-276-6040 Department of Electrical, Computer, and Systems Engineering, Rensselaer ...»

-- [ Page 1 ] --

Robust Real-Time Eye Detection and

Tracking Under Variable Lighting Conditions

and Various Face Orientations

Zhiwei Zhu a, Qiang Ji b

a E-mail:zhuz@rpi.edu

Telephone: 1-518-276-6040

Department of Electrical, Computer, and Systems Engineering,

Rensselaer Polytechnic Institute

JEC 6219, Troy, NY 12180-3590

b E-mail:


Telephone: 1-518-276-6440

Department of Electrical, Computer, and Systems Engineering,

Rensselaer Polytechnic Institute JEC 6044, Troy, NY 12180-3590 Abstract Most eye trackers based on active IR illumination require distinctive bright pupil effect to work well. However, due to a variety of factors such as eye closure, eye occlusion, and external illumination interference, pupils are not bright enough for these methods to work well. This tends to significantly limit their scope of application.

In this paper, we present an integrated eye tracker to overcome these limitations.

By combining the latest technologies in appearance-based object recognition and tracking with active IR illumination, our eye tracker can robustly track eyes under variable and realistic lighting conditions and under various face orientations.

In addition, our integrated eye tracker is able to handle occlusion, glasses, and to simultaneously track multiple people with different distances and poses to the camera. Results from an extensive experiment shows a significant improvement of our technique over existing eye tracking techniques.

Key words: Eye Tracking, Support Vector Machine, Mean Shift, Kalman Filtering Preprint submitted to Elsevier Science 5 July 2004 1 Introduction As one of the salient features of the human face, human eyes play an important role in face detection, face recognition and facial expression analysis.

Robust non-intrusive eye detection and tracking is a crucial step for vision based man-machine interaction technology to be widely accepted in common environments such as homes and offices. Eye tracking has also found applications in other areas including monitoring human vigilance [1], gaze-contingent smart graphics [2], and assisting people with disability. The existing work in eye detection and tracking can be classified into two categories: traditional image based passive approaches and the active IR based approaches. The former approaches detect eyes based on the unique intensity distribution or shape of the eyes. The underlying assumption is that the eyes appear different from the rest of the face both in shape and intensity. Eyes can be detected and tracked based on exploiting these differences. The active IR based approach, on the other hand, exploits the spectral (reflective) properties of pupils under near IR illumination to produce the bright/dark pupil effect. Eye detection and tracking is accomplished by detecting and tracking pupils.

The traditional methods can be broadly classified into three categories: template based methods [3–9,8,10,11], appearance based methods [12–14] and feature based methods [15–23]. In the template based methods, a generic eye model, based on the eye shape, is designed first. Template matching is then used to search the image for the eyes. Nixon [10] proposed an approach for accurate measurement of eye spacing using Hough transform. The eye is modeled by a circle for the iris and a “tailored” ellipse for the sclera boundary.

Their method, however, is time-consuming, needs a high contrast eye image, and only works with frontal faces. Deformable templates are commonly used [3–5]. First, an eye model, which is allowed to translate, rotate and deform to fit the best representation of the eye shape in the image, is designed. Then, the eye position can be obtained through a recursive process in an energy minimization sense. While this method can detect eyes accurately, it requires the eye model be properly initialized near the eyes. Furthermore, it is computationally expensive, and requires good image contrast for the method to converge correctly.

The appearance based methods [12],[13], [14] detect eyes based on their photometric appearance. These methods usually need to collect a large amount of training data, representing the eyes of different subjects, under different face orientations, and under different illumination conditions. These data are used to train a classifier such as a neural network or the Support Vector Machine and detection is achieved via classification. In [12], Pentland et al. extended the eigenface technique to the description and coding of facial features, yielding eigeneyes, eigennoses and eigenmouths. For eye detection, they extracted an appropriate eye templates for training and constructed a principal component projective space called “Eigeneyes”. Eye detection is accomplished by comparing a query image with an eye image in the eigeneyes space. Huang et al. [13] also employed the eigeneyes to perform initial eye positions detection. Huang et al. [14] presented a method to represent eye image using wavelets and to perform eye detection using RBF NN classifier. Reinders et al. [21] proposed several improvements on the neural network based eye detector. The trained neural network eye detector can detect rotated or scaled eyes under different lighting conditions. But it is trained for the frontal view face image only.

Feature based methods explore the characteristics (such as edge and intensity of iris, the color distributions of the sclera and the flesh) of the eyes to identify some distinctive features around the eyes. Kawato et al [16] proposed a feature based method for eyes detection and tracking. Instead of detecting eyes, they propose to detect the point between two eyes. The authors believe the point is more stable and easier to detect than the eyes. Eyes are subsequently detected as two dark parts, symmetrically located on each side of the between-eyepoint. Feng et al. [8,9] designed a new eye model consisting of six landmarks (eye corner points). Their technique first locates the eye landmarks based on the variance projection function (VPF) and the located landmarks are then employed to guide the eye detection. Experiment shows their method will fail if the eye is closed or partially occluded by hair or face orientation. In addition, their technique may mistake eyebrows for eyes. Tian et al. [19] proposed a new method to track the eye and recover the eye parameters. The method requires to manually initialize the eye model in the first frame. The eye’s inner corner and eyelids are tracked using a modified version of the Lucas-Kanade tracking algorithm [24]. The edge and intensity of iris are used to extract the shape information of the eye. Their method, however, requires a high contrast image to detect and track eye corners and to obtain a good edge image.

In summary, the traditional image based eye tracking approaches detect and track the eyes by exploiting eyes’ differences in appearance and shape from the rest of the face. The special characteristics of the eye such as dark pupil, white sclera, circular iris, eye corners, eye shape, etc. are utilized to distinguish the human eye from other objects. But due to eye closure, eye occlusion, variability in scale and location, different lighting conditions, and face orientations, these differences will often diminish or even disappear. Wavelet filtering [25,26] has been commonly used in computer vision to reduce illumination effect by removing subbands sensitive to illumination change. But it only works under slight illumination variation. Illumination variation for eye tracking applications could be significant. Hence, the eye image will not look much different in appearance or shape from the rest of the face, and the traditional image based approaches can not work very well, especially for faces with non-frontal orientations, under different illuminations, and for different subjects.

Eye detection and tracking based on the active remote IR illumination is a simple yet effective approach. It exploits the spectral (reflective) properties of the pupil under near IR illumination. Numerous techniques [27–31,1] have been developed based on this principle, including some commercial eye trackers [32,33]. They all rely on an active IR light source to produce the dark or bright pupil effects. Ebisawa et al. [27] generate the bright/dark pupil images based on a differential lighting scheme using two IR light sources (on and off camera axis). The eye can be tracked effectively by tracking the bright pupils in the difference image resulting from subtracting the dark pupil image from the bright pupil image. Later in [28], they further improved their method by using pupil brightness stabilization to eliminate the glass reflection. Morimoto et al.

[29] also utilize the differential lighting scheme to generate the bright/dark pupil images, and pupil detection is done after thresholding the difference image. A larger temporal support is used to reduce artifacts caused mostly by head motion, and geometric constraints are used to group the pupils.

Most of these methods require distinctive bright/dark pupil effect to work well. The success of such a system strongly depends on the brightness and size of the pupils, which are often affected by several factors including eye closure, eye occlusion due to face rotation, external illumination interferences, and the distances of the subjects to the camera. Figures 1 and 2 summarize different conditions under which the pupils may not appear very bright or even disappear. These conditions include eye closure (Figure 1 (a)) and oblique face orientations (Figure 1 (b) (c) and (d)), presence of other bright objects (due to either eye glasses glares or motion) as shown in Figure 2 (a) and (b), and external illumination interference as shown in Figure 2 (c).

–  –  –

The absence of the bright pupils or even weak pupil intensity poses serious problems to the existing eye tracking methods using IR for they all require relatively stable lighting conditions, users close to the camera, small out-ofplane face rotations, and open and un-occluded eyes. These conditions impose (a) (b) (c) Fig. 2. (a)original image, (b)the corresponding thresholded difference image, which contains other bright regions around the real pupil blobs due to either eye glasses glares and rapid head motion, (c) Weak pupils intensity due to strong external illumination interference.

serious restrictions on the part of their systems as well as on the user, and therefore limit their application scope. Realistically, however, lighting can be variable in many application domains, the natural movement of head often involves out-of-plane rotation, eye closures due to blinking and winking are physiological necessities for humans. Furthermore, thick eye glasses tend to disturb the infrared light so much that the pupils appear very weak. It is therefore very important for the eye tracking system to be able to robustly and accurately track eyes under these conditions as well.

To alleviate some of these problems, Ebisawa [28] proposed an image difference method based on two light sources to perform pupil detection under various lighting conditions. The background can be eliminated using the image difference method and the pupils can be easily detected by setting the threshold as low as possible in the difference image. They also proposed an ad hoc algorithm for eliminating the glares on the glasses, based on thresholding and morphological operations. However, the automatic determination of the threshold and the structure element size for morphological operations is difficult; and the threshold value cannot be set as low as possible considering the efficiency of the algorithm. Also, eliminating the noise blobs just according to their sizes is not enough. Haro[31] proposed to perform pupil tracking based on combining eye appearance, the bright pupil effect, and motion characteristics so that pupils can be separated from other equally bright objects in the scene. To do so, they proposed to verify the pupil blobs using conventional appearance based matching method and the motion characteristics of the eyes.

But their method can not track the closed or occluded eyes or eyes with weak pupil intensity due to external illuminations interference. Ji et al. [1] proposed a real time subtraction and a special filter to eliminate the external light interferences. But their technique fails to track the closed/occluded eyes. To handle the presence of other bright objects, their method performs pupil verification based on the shape and size of pupil blobs to eliminate spurious pupils blobs.

But usually, spurious blobs have similar shape and size to those of the pupil blobs as shown in Figure 2 and make it difficult to distinguish the real pupil blobs from the noise blobs based on only shape and size.

In this paper, we propose a real-time robust method for eye tracking under variable lighting conditions and face orientations, based on combining the appearance-based methods and the active IR illumination approach. Combining the respective strengths of different complementary techniques and overcoming their shortcomings, the proposed method uses an active infrared illumination to brighten subject’s faces to produce the bright pupil effect. The bright pupil effect and the appearance of eyes are utilized simultaneously for eyes detection and tracking. The latest technologies in pattern classification recognition (the Support Vector Machine) and in object tracking (the meanshift) are employed for pupil detection and tracking based on eyes appearance.

Some of the ideas presented in this paper have been briefly reported in [34],[35].

In this paper, we report our algorithm in details.

Pages:   || 2 | 3 | 4 |

Similar works:

«Sermon #2118 Metropolitan Tabernacle Pulpit 1 THE PLANTER OF THE EAR MUST HEAR NO. 2118 A SERMON INTENDED FOR READING ON LORD’S-DAY, DECEMBER 15, 1889. DELIVERED BY C. H. SPURGEON, AT THE METROPOLITAN TABERNACLE, NEWINGTON, ON THURSDAY EVENING, OCTOBER 31, 1889. “He that planted the ear, shall He not hear? He that formed the eye, shall He not see?” Psalm 94:9 THE character of a man hinges upon his relation to God. You may know what manner of man he is, and what his communications are, if...»

«THE CHANGING STRUCTURE OF THE HARDWOOD LUMBER INDUSTRY WITH IMPLICATIONS ON TECHNOLOGY ADAPTATION William Luppold John Baumgras USDA Forest Service Northeastern Research Station 241 Mercer Springs Road Princeton, WV 24740 ABSTRACT The hardwood sawmilling industry has been changing over the last 50 years as a result of changes in hardwood sawtimber inventory and in the demand for hardwood lumber. In 1950 the industry was composed of numerous individual mills, few of which produced more than 3...»

«MINUTES OF THE MEETING OF THE AEHT EXECUTIVE BOARD HELD IN LONDON (UK) ON MARCH 11TH, 2016 Members of the Executive Board together with the chief judges and the organising team of the 29th AEHT Annual Conference gathered on the docks next to the Docklands Academy London (DAL) COUNTRY / REPRES. PAYS First name / Name / Nom Prénom AEHT HEAD OFFICE Nadine SCHINTGEN SIEGE DE L’AEHT BELGIUM Herman SIEBENS BELGIQUE CROATIA Vesna BARANASIC CROATIE ESTONIA Neeme RAND ESTONIE FINLAND Marit NIEMINEN...»

«KERESAN PUEBLOS AND CHACO CANYON Speculations on Language Identity, Puebloan Cultures, and Southwestern Prehistory Jay Miller, Ph. D. Native Communities of the Southwest KERES (Keresic) TANOANS YUMANS Cochiti TEWA Quechan Santo Domingo San Juan Cocopa San Felipe Santa Clara Mohave Santa Ana San Ildefonso Maricopa Sia Pojoaque Walapai Acoma Nambe Yavapai Laguna Tesuque Havasupai TANO Hano (at Hopi) PIMANS TIWA Pima (O'odam) Taos Papago (Tohono O'odam) Picuris Sandia ATHAPASKANS Isleta NAVAHO...»

«SECURITIES AND EXCHANGE COMMISSION WASHINGTON, DC 20549 FORM 10-K ⌧ Annual report pursuant to Section 13 or 15(d) of the Securities Exchange Act of 1934 For the fiscal year ended December 31, 2003, or Transition report pursuant to Section 13 or 15(d) of the Securities Exchange Act of 1934 For the transition period from to. Commission File Number 1-9645 CLEAR CHANNEL COMMUNICATIONS, INC. (Exact name of registrant as specified in its charter) Texas 74-1787539 (State of Incorporation) (I.R.S....»

«Application Report SPRAA96A – April 2007 Encode Demo for the DVEVM/DVSDK 1.2 Niclas Anderberg SDO Applications ABSTRACT The DaVinci Digital Video Evaluation Module (DVEVM) comes with demonstration applications that illustrate the use of its software and hardware components. This document describes the design of the “encode” demo application. The encode demo uses the Codec Engine as well as video and speech algorithms from Texas Instruments to encode video and sound data from peripheral...»

«Otherness Essays and Studies 3.2 Bildungsroman of the Bastard: Dorothy Allison and the Genealogy of Illegitimacy Linda Camarasana Dorothy Allison‟s Bastard out of Carolina is a novelwhose very title, hero, and subject are illegitimatethat reveals the entanglements between various systems of surveillance, and how these systems inscribe class, gender, racial, and sexual illegitimacy in the novel. The first chapter of Dorothy Allison‟s quasiautobiographical bildungsroman reveals the immediate...»


«About SLP SLP’s stainless steel catback exhaust system, Providing OEM-quality performance, handling and SLP’s high-flow intake runners, SLP’s cold air appearance components for late model Detroitinduction system and an SLP revised PCM E-Prom. based Performance Cars (Camaro, Corvette, Installing this package on a TPI 350 CID engineMustang, Challenger and Charger) and Full-Size GM equipped 1988 Camaro resulted in a 50 horsepower Trucks and SUVs is what SLP is all about. gain, a one full...»

«Baalism, Eco-Fascism & the Apocalypse Psalm 2 Why do the Heathen Rage? ŸWb’C.y:t.yIï 2 `qyrI)-WGh.y ~yMiªaul.W÷ ~yI+Ag Wvågr hM'l'â `Ax*yvim.-l[;w hwhy÷-l[; 3 dx;y+-Wds.An* ~ynIïzArw #ra,ªykel.m; bveäAy 4 `Amyte(bo[] WNM,ämi hk'yliÞv.n:w Amyte_Ars.Am)-ta, hq'T.n:nâ) AP+a;b. Amyleäae rBEåd:y za'Û 5 `Aml'(-g[;l.yI yn©doa]÷ qx'_f.yI ~yIm:åV'B; `yvi(dq'-rh; !AY©ci÷-l[; yKi_l.m; 7 yTik.s;än ynIa]w:â 6 `Amle(h]b;y Psalm 2 Summary • 2:1-3 – David describes the...»

«UNA PROPUESTA DE TRABAJOS PRÁCTICOS DE LABORATORIO QUE FAVORECE EL APRENDIZAJE DE CONCEPTOS A proposal for laboratory practice in favor of concept learning Marisol Montino1 José Ernesto Ure3 Diego Petrucci2 Alejandra Aleman4 Silvia Margarita Pérez5 Resumen: Se presenta el análisis y discusión de la implementación de una modalidad de trabajo práctico de laboratorio (TPL) para el nivel universitario, desarrollada a partir de los resultados de investigaciones propias. El formato pretende...»

«MONA SINGH ON THE SEMANTICS OF THE PERFECTIVE ASPECT* The study of the temporal structure of events in natural language is of prime importance in linguistics. Though there has been recent progress on formal theories of events, these theories do not address certain syntactic and semantic properties peculiar to languages such as Hindi. This paper concentrates on properties related to perfectivity. It motivates a small number of semantic features for events and their objects, such as whether an...»

<<  HOME   |    CONTACTS
2016 www.dissertation.xlibx.info - Dissertations, online materials

Materials of this site are available for review, all rights belong to their respective owners.
If you do not agree with the fact that your material is placed on this site, please, email us, we will within 1-2 business days delete him.