Working for People with Sight Loss

App in Focus – Microsoft Seeing AI

Microsoft App open on iPhone showing photo of girl at a restaurant table smiling

Microsoft App open on iPhone showing photo of girl at a restaurant table smiling

Seeing AI is a free app available from the Apple App Store. Microsoft describes this app as a Microsoft research project for vision impaired people.

The app narrates the world around you by turning the visual world into an audible experience. That is their long-term plan as part of this project. In the meantime, they have released a great app with some great features which we will mention below in order of usefulness.

When you open the App for the first time it will give you an option to watch or listen to some video tutorials on all the different channels available in the app. It is a good idea to familiarise yourself with how to use the App before you start. When you are finished watching the tutorial videos you can navigate your way through the different channels by flicking right until you hear “channel, short text adjustable” now you can arrow up and down through the different channels.

Short text

The most used part of this app is a feature called short text. This allows you to use your phone’s camera over a piece of text and then it will start to read it out to you, and as new text comes into view it will also be read out loud. It can be useful if you want to read your computer screen if your screen reader has gone quiet, or for quickly reading a road sign, or to find out the name of a product.

Document OCR

Hold the camera over a printed page to capture it. You must hold the camera approximately 12 inches above the document to get the full document into the cameras field of view. Seeing AI will guide you with camera placement until all edges of the document are visible and the photo is taken. This channel works best when you have a good contrast between the document and the background, and you have good lighting. Once the text has been recognised you can read the text with voiceover and change the size, save, or share it.

Product/ barcode scanner 

You must hold your phone camera over the barcode of any product. Seeing AI will guide you with the camera placement until a barcode is detected. When a barcode is detected Seeing AI will say the product name and may suggest other instructions if available such as cooking instructions.

Currency

Seeing AI can also recognise currency.

Hold the camera over a single note to hear its value. This works well, and it also recognises the note very quickly.

Light

Use this channel to detect the amount of light around you. It might be useful for blind person to find out if the lights are left on or not. The pitch of the tone is based on how much light your phone sees, the more light there is the higher the pitch of the tone.

Colour

Seeing AI can also be used to detect colour. Note this is not always accurate and can depend on the lighting in the room or if a shadow is present but in general it works well and can be useful in some cases.

Seeing AI has also a few experimental channels built in for you to try. These include:

Handwriting

If Someone gives you a handwritten note or even a Birthday card you can try and read it with Seeing AI in the same way as you use the document OCR Channel. Note for this feature to work it requires you to have the handwriting the right way up. Results may vary depending on people’s different styles of handwriting.

Scene Preview

Another experimental feature is scene preview. Hold your camera up and take a picture of the area around you and using Artificial Intelligence voiceover will describe the scene or area that was captured.

Person

The last feature we will mention is Seeing AI’s ability to recognise people.  You can scan your surroundings with your phones camera to find out how many people are around you. It will also tell you the position of each face it detects, which can be helpful when centring a person in the cameras view when taking a photo. After taking a photo Seeing AI will give you an estimate of the persons facial characteristics and expressions and will even try and guess their age. This is a fun feature and if you want to take a selfie use the button on the main screen to change the camera to front facing camera.