Recently I created a project for a Swedish Kinect contest at Migbi.se this was my second entry, my first one (and third place winner) was my Robosapien project.

I have always been fascinated by home automation, I bought my first X-10 system ten years ago.

My friends thought I was insane, “-You can just get up an shut the light off".

But that’s not the point, it’s not because I’m lazy it’s all about removing obstacles, what if when I enter a room the lights turns on, when I go to bed everything turns off.

Saves energy, saves time, removes obstacles.

 

Peter Forss made a really cool entry to the contest, his project turns on and off lights depending on where he is in the room.

This inspired me, I wanted to do something with Kinect and home automation.

 

I had previously built a home automation system that can control our home (lights, infrared devices etc) so the only thing I needed to do is hook up the Kinect.
I wanted to be able to control what lights to turn on just by pointing at them.

So here is my attempt to control our living room with a Kinect.

 

 

How it works

I added all my light in an array with the lights X and Z position relative to the kinect sensor (in meters).

For each light, I calculate the angle from where I am in the room to the light and compare it to the angle between me (my body’s centre) and my hand.

 

Then I check for the light on gesture (hand under shoulder moved to over shoulder) or light off gesture (hand over shoulder moved to below shoulder).

 

These gestures sends a command to my home automation system to executes the correct command.

It uses a Tellstick to control the lights, the beauty of that device is that it can control close to any type of protocol (I use Nexa or in some cases the cheapest possible plug-in lamp module I could find, it also works with X10).

 

In this video I only control lights and screen, but it is possible to control infrared devices like tv or home cinema.

 

Please feel free to send me an email if you have any questions.

First off I would like to start with saying that the title on this blog post might be just a little exaggerated.

The Gadgeteer is running C# code that emulates the ZX Spectrum and so far it’s true.

Basically the emulator works by running cycles, showing the screen and repeating.

To get the correct timing this should be done 50 times per second, which means that we have 20ms to complete one cycle and show the screen.

The Gadgeteer is far from fast enough to achieve that, right now it takes 10 seconds to complete a cycle.

But this wasn’t the point, I suspected it wouldn’t be fast enough. The point was that is was possible \o/.
Within hours I managed to connect a screen to the Gadgeteer and make the necessary changes in the code to make it run.

For example I had to change List<> to an array, the .NET Micro framework doesn’t support List<>, and I rewrote the screen rendering to make it faster.

 

I find it fantastic that I can use my C# knowledge to create new hardware prototypes among all the other things like: xbox games, Windows Phone applications and games, Windows applications and games, and even write Iphone and Android applications.

There is no end to the possibilities =)

 

Nuance recently released a windows phone 7 SDK for their text to speech and dictation services.

I have been looking for some kind of text to speech that can handle Swedish and also being able to control things with voice commands in Swedish.

I noticed that Nuance supported that so I decided to sign up as a developer.

Windows Phone 7 already supports TTS for reading sms and also some voice control for searching and opening applications but only support the major languages (Swedish not included).

I have an application idea for the Swedish market that could use voice control (no I’m not saying what it is ).

The SDK includes some sample code that makes it easy to get started.

What I didn’t find anywhere was instructions on how to get this working for languages other than English, and I couldn’t read the help files for some reason.

 

So here is what you need to do:

For dictation support: Replace all the _oemconfig.defaultLanguage() with a string containing your preferred language (sv_SE for Swedish).

For TTS: Add a voice that supports your language (Alva for Swedish)

That is it, now you can play with the app.

I think it works ok, but not as good as I hoped.

 

NDEV Mobile

http://dragonmobile.nuancemobiledeveloper.com/public/index.php

 

Voices (this page is for another product but seems to be the same as the Mobile SDK

http://www.nuance.com/vocalizer5/languages/

Got my Gadgeteer compatible FEZ Spider kit a couple of days ago.

For those not familiar with Gadgeteer, it is an open source toolkit for building small electronic devices using .NET Micros Framework.
Microsoft is behind this project and GHI is the first manufacturer to release compatible hardware.

I have used Netduino before, the “problem” with Netduino is that it is extremely basic, you really need to know electronics.

Since I read a lot of these things in school I thought it would be a piece of cake getting stared with Netduino but reality soon came knocking.

Apparently nearly everything I learned in school about these things are gone, quite easy to read up on but still an obstacle between me and my new hardware toy .

Gadgeteer is a fast prototyping kit that uses standard 10 pin sockets that are impossible to turn the wrong way.
It makes it really simple and removes obstacles between you and your finished prototype.
What I like about the Gadgeteer is that it has very little integrated functions, if you want Ethernet for example you need to connect the Ethernet module, there is not Ethernet built-in on the motherboard.

It has 14 connectors and every connector has a letter beside it.
The letter indicates what kind of module you can connect to that socket.

 

Installation

Its really simple to get started, GHI has an install-package that include everything you need to make your first application, or perhaps gadget is a more accurate description.

 

Start Coding

Now you are ready to start your first Gadgeteer project.

In Visual Studio you will find a new template section called Gadgeteer, to start making an app simply select “.NET Gadgeteer Application”.

What I saw next really made me happy, in Visual Studio you’ll get a designer window.

You only need to drag in the modules you wish to use and connect them to a compatible socket.

The designer even tells you which sockets that are ok.

But it doesn’t stop there!
Instead of connecting them manually you can right click and select “connect all modules” and the designer does the job for you.

These kind of help functions really makes me happy, I want to concentrate on coding not other things.

 

Some initial problems
I started of by adding a Multi color LED and wrote
led.TurnBlue();

And the LED did just that, it turned Green.. Say what now!?
Apparently there is a bug, in the LED’s firmware, but GHI is working on the problem.

Next I tried to make a camera app, press a button, take a picture and show it on the screen.

However the buttons didn’t trigger the pressed event this is also an known issue.

So the first two things I tried fail which actually made me a bit concerned.

GHI seems to be on top of things and is really active on their forums so I hope they will come up with a great solution.

Despite the initial problems I would say, buy this kit =)
There is a special feeling when you code runs on a device (like a phone for example).
There's an even more special feeling when your code runs on a device you just put together =)

What does the Motion API do?

Well basically it combines the different sensors into one sensor.

 

Accelerometer + Gyroscope
The accelerometer can tell the orientation of the phone by measuring the gravitational forces.
The gyroscope measures the changes in the orientation which means when you start the gyro it won’t be able to tell you what orientation the phone has but it can tell when and how much the orientation changes.
The motion API combines these two values to get the phone's orientation with the accelerometers and uses that as a baseline and then adjust these values with changes from the Gyroscope.
The gyro has a tendency to drift over time so the accelerometer will be used to adjust the gyroscope from time to time.

 

Compass + Gyroscope
The compass is a slow sensor and is used the same way as the accelerometer as a baseline for the gyro.

The Motion API uses the compass to get the heading (North) and then uses the gyro changes to adjust the bearing.
Since the gyro has a tendency to drift (as I mentioned before) the compass will be used to adjust these values over time.

The compass is also very sensitive to magnetic fields and jumps around a lot, the gyro will smooth these readings out.

 

The gyro do not contribute with any “new” values, the accelerometer and compass will do just fine when it come to retrieving the phone's orientation and which way it is facing however it will speed up these readings and make them a whole lot smoother.

The gyro is also more precise for smaller movements.

 

I’m definitely seeing to that my next phone will have a gyroscope, it makes the augmented reality experience a whole lot better.

Mango introduces a new way to show directions called BingMapsDirectionsTask.

The usage is very simple:

 

using Microsoft.Phone.Tasks;

BingMapsDirectionsTask directions = new BingMapsDirectionsTask();
directions.Start= new LabeledMapLocation("start",new System.Device.Location.GeoCoordinate(59.3362,18.0710);
directions.End = new LabeledMapLocation("stop",new System.Device.Location.GeoCoordinate(59.3360, 18.0679);
directions.Show();

Quite simple huh? But there is a catch, the above code won’t work if you are using a regions format that uses comma instead of dot as a decimal separator.
Swedish is one of those. I have tried some different ways of solving that by doing a reverse geocoding with the bing services but it didn’t help.
I haven’t done a deep analysis of this but what I think happens is that the BingMapsDirectionsTask wont recognize the GeoCoordinates as a correct point and then uses the text I have supplied (in this case "start" and "stop") to try to find the address instead. But in this demo I don’t have the correct address, and IF I did it wouldn’t understand the special character of the swedish language (åäö) which will result in a box asking which address is the correct one, even though I know the correct GPS coordinates.

I don’t want to ask the user, the user won't know.
My guess is that Microsoft will fix this soon, but perhaps not in Mango when it releases.

What can we do? There is a quite simple solution to solve this. Before you call the directions.Show() you can set the current culture to en-US and set it back.
This is how I solved it:

string realCulture = System.Threading.Thread.CurrentThread.CurrentCulture.Name;
try
{
       BingMapsDirectionsTask directions = new BingMapsDirectionsTask();
    directions.Start= new new LabeledMapLocation("start",new System.Device.Location.GeoCoordinate(59.3362,18.0710);     directions.End = new LabeledMapLocation("stop",new System.Device.Location.GeoCoordinate(59.3360, 18.0679);
    Thread.CurrentThread.CurrentCulture = new CultureInfo("en-US");
    directions.Show();
}
catch { }
finally
{
    Thread.CurrentThread.CurrentCulture = new CultureInfo(realCulture);
}

It will however still have problems with Swedish characters in LabeledMapLocation labels so you might want to avoid to supply special characters. In this particular case the important thing to me is that the direction task is working.

I hade some difficulties installing Windows 8 Developer preview on my machine.

The setup complained about cd/dvd driver is missing, after some binging I found that the same error in Windows 7 was a result of a corrupt ISO, this did however not apply for me, my checksum was correct.

Since I wanted to rule out a burning issue I created a bootable USB-stick by using Windows 7 USB/DVD download tool found on codeplex.
This didn’t help still got the same error.

So what I ended up doing was installing Windows 8 from within Windows 7 (already installed on my laptop) (Really nice experience).

 

When the installation was done I noticed that the touch worked great, really responsive but two things didn’t work: Tap and Flick.

I downloaded the Windows 7 n-trig drivers and tried to install them but the installer doesn’t work since I didn’t have Windows 7 (the installer checked for that).

Using an application called Universal Extractor I unpacked the installation files and the ran DPInst as an administrator.

After that the touch experience works just as expected.

I find Windows 8 a lot faster than Windows 7 it gives just that little extra an old tired laptop needs.

 

If you have any questions feel free to email me.

 

Resources

Windows 8 developer preview

Website

 

Universal extractor

Website

 

Windows 7 USB/DVD download tool (possibly not needed)

Website

 

Windows 7 n-trig drivers

Website

Augmented reality has really caught my eye, I just love how the computer generated items interact with the real world.

I have played with the SLAR toolkit which is a really awesome library, but its a little bit limited in what it can do since you need a marker to get it to work.

 

How about augmented reality without markers?
In mango, developers are given access to two necessary components to make augmented reality a ehm.. reality.

Raw camera access (So we can show what the camera sees)
Compass (access to the compass)

With mango we also get a new sensor, the gyroscope (optional on mango phones).

 

When I first tried using the compass it didn’t work, that’s because I was running the Mango Beta supplied by Microsoft.

The drivers for the compass is not included in that build, it is supplied by the phone manufacturer later on in the process.

 

Let’s go through the available sensors

Accelerometer

An accelerometer measures acceleration forces. These could be static like measuring the constant pulling of gravity or more dynamic like moving (sudden starts and stops) or vibrating the accelerometer.

The normal use for an accelerometer is sensing if the phone is tilted (portrait or landscape) but can also be uses for example in games to steer something.

This sensor is available on all devices (including pre mango)

 

Compass

A compass (or magnetometer) measures the strength and/or direction of magnetic fields.

Using the compass we can get our bearing relative to the geographic north.

Keep in mind that the Earths magnetic field are relatively weak and the compass will be easily manipulated by other fields.

To test an app using the compass I would advice you to go outside, inside there is much disturbance (in the force.. oh I mean … the magnetic fields).

The compass is optional in phones.

 

Gyroscope

The gyroscope measures orientation changes, it will give you a more accurate measurement of how your device orientation has changed.

The gyro does not depend on gravity like the accelerometer, and unlike the accelerometer it will enable us to measure for example if you rotate your phone while placed on a flat surface.

Optional on Mango phones and missing on pre-mango.

 

 

So how do we use this?

Microsoft has provided us with the Motion API which takes the available sensors and some magic to help us get more accurate and faster readings.

For the Motion API to be enabled you need at least Accelerometer and Compass (so you will be able to use it on pre-mango devices \o/ ).

 

Here is my first video of an augmented reality app.
Its based on the code provided in the videos below in the resources section.

I also added a semi-transparent Bing map on top that the user can show or hide.

 

The videos below does a really good job describing how the Motion API works so I won’t get into any details for now.

 

Resources
Great  video explaining how things work
http://vimeo.com/27378156

Another great video to get started with augmented reality (with source code)

http://vimeo.com/27377090

In my last augmented reality demo I showed a 3d model of a ZX Spectrum that appeared on a marker.

I got a few request on sharing the source so I cleaned up the code a little bit and posted the source for a version of that demo.

But let’s face it a 3d model on a piece of paper is cool but is there a way to make it even more exciting?

As you may or may not know I’m working on a ZX Spectrum emulator for Windows phone 7 and my goal is to have it ready very close to the mango release.

Perhaps integrate the two? How about an augmented reality ZX Spectrum emulator?

 

 

What amazes me is the performance of these phones, I have the Samsung Omnia 7 and it manages detecting the marker, showing the camera feed, rendering the 3d model and emulating a ZX Spectrum with sound and all at 25 fps.

I’m really impressed, imagine what we can do with compass and gyro.

 

Got a question if I was willing to share the code for my augmented reality app.

There are some samples out there but they’re often trying to show off how insanely cool augmented reality is and has loads of code that can be hard to get thru and understand.

I don’t think I need to explain the code in this post, I have commented in the code, I hope it will be easy to understand.

In my app I used a Silverlight / XNA combination, that is probably a bit overkill only for this but for those not familiar to XNA it might be easier to read so I’ll go with the same approach here.

You’ll need to download the source code from codeplex for SLARToolkit since the current release doesn’t include everything you need.

For simplicity my sample code also includes the necessary SLARToolkit code (at the time of writing this Changeset 67779).

 

There is something wrong when navigating from the game page and then back again, haven't been able to find what yet, if you have any suggestions please send me an email.

 

If you have any suggestions on how to make the code better again please contact me I love to learn new things =)

 

AugmentedRealityDemo.zip (2.34 mb)