«Augmented Reality Applied to Language Translation Dissertação para obtenção do Grau de Mestre em Engenharia Electrotécnica e de Computadores ...»
Ana Rita de Tróia Salvado
Licenciado em Ciências da
Engenharia Electrotécnica e de Computadores
Augmented Reality Applied to
Dissertação para obtenção do Grau de Mestre em
Engenharia Electrotécnica e de Computadores
Orientador : Prof. Dr. José António Barata de Oliveira,
Prof. Auxiliar, Universidade Nova de Lisboa
Presidente: Doutor João Paulo Branquinho Pimentão, FCT/UNL
Arguente: Doutor Tiago Oliveira Machado de Figueiredo Cardoso, FCT/UNL Vogal: Doutor José António Barata de Oliveira, FCT/UNL September, 2015 iii Augmented Reality Applied to Language Translation Copyright c Ana Rita de Tróia Salvado, Faculdade de Ciências e Tecnologia, Universi- dade Nova de Lisboa A Faculdade de Ciências e Tecnologia e a Universidade Nova de Lisboa têm o direito, perpétuo e sem limites geográﬁcos, de arquivar e publicar esta dissertação através de ex- emplares impressos reproduzidos em papel ou de forma digital, ou por qualquer outro meio conhecido ou que venha a ser inventado, e de a divulgar através de repositórios cientíﬁcos e de admitir a sua cópia e distribuição com objectivos educacionais ou de in- vestigação, não comerciais, desde que seja dado crédito ao autor e editor.
iv To my beloved family...
vi Acknowledgements "Coming together is a beginning; keeping together is progress; working together is success." - Henry Ford.
Life can only be truly enjoyed when people get together to create and share moments and memories. Greatness can be easily achieved by working together and being sup- ported by others. For this reason, I would like to save a special place in this work to thank people who were there and supported me during all this learning process.
I would like to thank Faculdade de Ciências e Tecnologias from Universidade Nova de Lisboa, for being my second home for the last ﬁve years. It was very important to ﬁnd a place where we have friends and people who care. Arriving early in the morning and listen to the silence, while watching the sea from a distance. Leaving late in the evening, while looking at the stars. Being, not only a student, but also the Student.
A special thank you to professor José Barata. Quoting Bill Gates, "Technology is just a tool. In terms of getting the kids working together and motivating them, the teacher is the most important." It was an honour to work with such a personality, who even introduced me to other bright minds that helped me through this experience. Thank you very much for believing and giving me the opportunity to have a personal touch when choosing the theme, as well as supporting me with knowledge and motivation.
Thank you to all the teachers, who went into my path with expertise and commitment. In particular, Professor Helena Fino: a woman in a men’s Department, a character, the Female Character from MIEEC. Thank you very much for assisting and guiding me throughout my journey. For being available every time I needed. Thank you for sharing wisdom, experience and life stories.
A special thank you to my colleagues in the robotics lab, who friendly welcomed me into their working space: Paulo Rodrigues, Ricardo Pombeiro, João Gomes, Francisco Marques, Ricardo Mendonça, André Lourenço and Eduardo Pinto. Thank you so much João Ramalho Carlos, for friendship and companionship. Thank you to my uni’s godfather and friend, Jorge Marques Silva, for help and guidance. Also, thank you to my
friends, who gave me a life beyond work, full of happiness and joy: Tiago Pereira, Fábio Nogueira, Fábio Lourenço, Bruno Dias, Celso Almeida, Tiago Antunes, Tiago Bento, António Bernardino, Cristiana Nóbrega, António Sá, Sara Ribeiro, Miguel Prego, Gisela Seixas, Rodrigo Francisco, Andreia Ribeiro, and so many others. And dear Mrs. Rosário Grancho, thank you for a smile every day I arrived at the Department.
Ponto Zero, my second family, thank you for bringing gymnastics into my education.
For the best moments of fun and craziness. To my coaches, João Martins and Pascoal Furtado, a huge thanks for being in my life.
Finally, thank you so much to my family and close friends: mom, dad, grandpas, godparents... and everyone who has helped and believed in me to achieve so much.
Dear Bia, thank you for being with me, even when messing with my patience, but always around with singing, playing, happiness and strength.
Abstract Being a tourist in a foreign country is an adventure full of memories and experiences, but it can be truly challenging when it comes to communication. Finding yourself in an unknown place, where all the road signs and guidelines have such different characters, may end up in a dead end or with some unexpected results. Then, what if we could use a smartphone to read that restaurant menu? Or even ﬁnd the right department in a mall?
The applications are so many and the market is ready to invest and give opportunities to creative and economic ideas.
The dissertation intends to explore the ﬁeld of Augmented Reality, while helping the user to enrich his view with information. Giving the ability to look around, detect the text in the surroundings and read its translation in our own dialect, is a great step to overcome language issues. Moreover, using smartphones at anyone’s reach, or wearing smartglasses that are even less intrusive, gives a chance to engage a complex matter in a daily routine.
This technology requires ﬂexible, accurate and fast Optical Character Recognition and Translation systems, in an Internet of Things scenery. Quality and precision is a must, yet to be further developed and improved. Entering in a realtime digital data environment, will support great causes and aid the progress and evolution of many intervention areas.
Keywords: Abbyy, Android, Augmented Reality (AR), Optical Character Recognition (OCR), Realtime, Smartglasses, Smartphones, Tesseract, Translation, Vuforia
Ser um turista num país estrangeiro é uma aventura cheia de memórias e experiências, mas que pode ser verdadeiramente desaﬁante em termos de comunicação. Encontrarse num lugar desconhecido, onde os sinais rodoviários e orientações têm tão diferentes carateres, pode acabar num beco sem saída ou com resultados inesperados. Então, e se pudéssemos usar o smartphone para ler aquele menu do restaurante? Ou até encontrar o local certo num centro comercial? As aplicações são tantas e o mercado está preparado para investir e dar oportunidades a ideias criativas e económicas.
A dissertação pretende explorar o campo da Realidade Aumentada, ajudando o utilizador a enriquecer a sua visão com informação. Permitir olhar em redor, detetar o texto à volta e ler a sua tradução no nosso próprio dialeto, é um grande passo para ultrapassar problemas de comunicação. Utilizar smartphones, ao alcance de qualquer um, ou usar smartglasses que são ainda menos intrusivos, dá a possibilidade de incorporar um assunto complexo na rotina do dia-a-dia.
Esta tecnologia requer sistemas de Reconhecimento Ótico de Carateres e Tradução ﬂexíveis, exatos e rápidos, num cenário de Internet das Coisas. Qualidade e precisão são uma necessidade, ainda por desenvolver e melhorar. Entrar num ambiente de dados digitais em tempo real irá apoiar grandes causas e auxiliar no progresso e evolução de muitas áreas de intervenção.
Palavras-chave: Abbyy, Android, Realidade Aumentada, Reconhecimento Ótico de Carateres, Smartglasses, Smartphones, Tempo Real, Tesseract, Tradução, Vuforia
ALPAC Automatic Language Processing Advisory Committee API Application Programming Interface App Application Software AR Augmented Reality ARTrS Augmented Reality TranSlation BARS Battleﬁeld Augmented Reality System CDT C/C++ Development Tooling GPS Global Positioning System HMD Head Mounted Device ICR Intelligent Character Recognition IDE Integrated Development Environment IMU Inertial Measurement Unit IoT Internet of Things JVM Java Virtual Machine MT Machine Translation NDK Native Development Kit NFT Natural Feature Tracking OCR Optical Character Recognition PoC Proof of Concept
PRONI Public Records Ofﬁce of Northern Ireland QCAR QualComm Augmented Reality RICS Robotic & Industrial Complex Systems ROI Region Of Interest SDK Software Development Kit UI User Interface
1.1 Context and Motivation The world is a wide place, full of ancient culture, where the people living in it feel the need to connect. Communication is necessary for human relations. However, there is no global language yet. There are about 2,000 idioms known on the planet, which can differ from one another through sound, written accent, reading or even completely different written characters.
Travelling is a hobby loved by many, but without proper communication, it may be hard to completely enjoy. Going to another country can be a true challenge when everyone around does not speak the same dialect. Natural acts, such as arriving at the airport and ﬁnding the exit, reading the signs in the street to reach a place, going to a restaurant and know what to ask for, communicating with the local people, they all may become overwhelming. These are just a few issues that a foreigner has to deal with, when going to another country. This is particularly relevant for people coming from regions where English, as a second language, is not well developed. For instance, people from Angola, when travelling to non Portuguese speaking countries, usually require to be with someone that speaks English.
Have you ever been to an exquisite foreign restaurant, all excited to try new food, until the waiter arrives with the menu, from which you see so many different options, with no images to connect the unknown names. There is no way you can choose the best dish without making a lot of questions. Sometimes, you even have to bet, as in a Russian roulette, to decide the winning dish. Having an application prepared to capture the unknown text and translating it to your own language would be a big help in situations like the ones referred before.
1. INTRODUCTION 1.2. Goal and ApproachNowadays technology is an every minute resource in life. Everything is becoming more connected. Every device, every equipment. Communication via wireless is making the world smaller. Smartphones have become an indispensable object, with many functionalities, used to help in numerous daily tasks. As a result of combining the previous facts, translation applications are a common target for many developers. Smartphones are currently widely disseminated, either in developed or undeveloped countries, which turns translation applications even more needed. Gadgets like smartglasses have been developed and launched into testing environments in order to keep up with modern and progress needs.
An approach to improve the new applications in a modern and smart way is by using Augmented Reality, a technology that makes reality richer and more interesting by merging layers of digital information with a device’s camera view ([Cra10]). This enables internet to be integrated within the user’s realtime environment. The ability to allow the customer to experience including data in his view perspective, will give him the illusion of entering into this new "Real Data World".
1.2 Goal and Approach The goal of this work is to allow people to recognize the information in their own language when travelling abroad. For this purpose, the approach would be implementing an application capable of detecting text in the user’s surroundings and immediately translate it to another language. Augmented Reality is a mean to achieve this goal in a creative, innovative and futuristic strategy.
The application may be prepared to run on different platforms. Programming for Android and iPhone devices offers better mobility and ﬂexibility to mobile applications, since it allows them to be accessed anywhere, any time. The development for smartphones makes the App accessible for different types of costumers. Smartglasses are also a target widget, so it is possible to explore and experiment the recent Augmented Reality ambition with the most modern equipment.
The project intends to look into techniques of Optical Character Recognition and analyse them when processing a photograph or displaying realtime results. The methods should be compared and tested with various sample images, to evaluate the Proof of Concept application’s versatility. A Machine Translator is also considered and implemented, so that an opinion can be built around this theme.
The results have to be suitable for using the PoC application in the real world. It should deal with natural adversities of the surroundings and perform a user friendly program, with fast and accurate output.
Robotic & Industrial Complex Systems (RICS) group is a team of researchers afﬁliated to Universidade Nova de Lisboa, that explore both Mobile Autonomous Robotics
Systems and Industrial & Intelligent Manufacturing Systems, further detailed at http:
//rics.uninova.pt/. This project is a research work with the purpose of exploring 2
1. INTRODUCTION 1.3. Dissertation Structurethe Augmented Reality concept while enhancing the RICS group with knowledge in this ﬁeld.
"ROBO-PARTNER" intends to lend a safer hand to the human in industrial automation systems. Also, "Self-Learning Production Systems" are being researched and developed to better ﬁt the needs from society. Imagine the advantages of being able to control the ROBO-PARTNER, while seeing information about the whole system dynamic that only a machine would see. Or even controlling an autonomous boat from a distance and seeing where he goes, in order to help its autonomous system in dangerous situations.
AR supported by the smartglasses can provide a revolutionary easy control upon these matters.
AR is a new technology rising great expectations, meant to be integrated in several developing projects. Therefore, this dissertation intends to analyse some methodologies to deal with smartglasses and the use of Augmented Reality.
1.3 Dissertation Structure This dissertation is organized in six chapters, along with this section, and one annex to
support the study: