Prologue

Posted on June 21, 2011 by Petri Partanen

(Updated: June 29, 2011)

During the University course TIEA343 - Robottiohjelmointi we had a change to build our own robot with a Lego NXT set. I received the NXT version 1.0, which has a light sensor instead to the color sensor supplied with NXT 2.0, and it also had the quite useless sound sensor.

After several different car models and other attempts at making "something", I ended up building an automatic panoramic camera bot. I mounted my not-so-recent digital camera to the robot and built a frame which rotated the camera almost all angles possible, and programmed the NXT to take pictures from different positions.

I also found an awesome progam called Hugin (cross-platform!), which did an impressive job at joining the pictures to a panoramic view. There was very little, if at all manual tinkering to get the images positioned correctly.

Panoramic Pictures

Panoramic picture is a photo which covers very wide angle of it's surroundings, usually entire 360 degrees. Most pictures you see will not include the picture of the sky above the camera, but I decided to include it anyways.

There are several ways of taking panoramic pictures, such as using fixed lenses or mirrors. I wanted to use a regular digital camera; so I will only talk about the rotating camera method from now on. Using rotations to take panoramic pictures results in severe distortion of straight lines, and images taken this way are best viewed in curved plane as opposite to flat images. However; rotation of the camera is probably the cheapest way to produce panoramic pictures.

The digital camera I had was a Canon PowerShot A590 IS. It is relatively old and has a very narrow lens, meaning that in order to take a panoramic picture; I was forced to take quite a lot of pictures to cover entire perspective.

The Robot

The robot has all three motors. One for horizontal rotation, one for vertical tilt and third motor for the camera shutter. Touch sensor is used to stop and reset the vertical angle during initialization procedure at startup. Light sensor is used to detect change in the digital camera's LCD screen during picture time.

For the horizontal rotation you can either turn only the camera, or you can turn the entire assembly around. I decided to go with the entire system rotation, this way there is no danger of dangling the wires and it adds some stability.

I did have some problems with the horizontal turning as I didn't quite choose the best gears for this purpose. So I had to re-inforce the assembly a bit to get some reliability, yet it still isn't very good. It does however work perfectly if you just re-check the Lego parts now and then as they become slightly loose over time.

The vertical rotation works well enough for it's purpose. The camera rack is slightly top heavy, which makes it's rotation slightly inaccurate, but this isn't really a problem as long as you take enough pictures from different vertical angles thus giving some overlap between them.

The "shutter's" motor is very simple and works well enough for this particular camera set-up. I could have easily used the bolt mounting on the bottom of the camera, which would have made the camera rack much more simple in design (and lighter!), but I didn't bother with it. All the parts are from NXT 1.0 set, there are no extra Lego parts or non-Lego parts, except the camera of course...

I did use a little piece of cardboard and tape to conceal the light sensor behind the LCD screen in order to eliminate any excess ambient light entering in. I'm not quite sure how important this step really was, but seeing how the robot sometimes turns around the sun there is a large change of sicnificant light changes, which could mess with the light sensor readings.

Programming

The robot was programmed with leJOS NXJ 0.90. I used the subsumption system for the robot's behaviour chain. Programming the robot was very easy, I excpected a lot more tinkering on the little things, but the leJOS pretty much does most of the hardware stuff; which was very nice.

The picture taking has higher behavior status than the rotation. Picture is always taken if no picture has been taken from the angle the robot currently is in (and the motors aren't moving). The light sensor gives a go signal for the picture taking behavior to terminate immediatily when it notices a picture has been taken. And of course rotation happens if no picture taking process was triggered and it just moves into the new angle (if any left).

The source code is provided in two files:

Javadoc is: here.

The Panoramio.java includes the main() and subroutines which ask user for settings during startup. main() creates a new Brains Object and initializes it, which then takes over the control. Brains.java is where the Brains Object is initialized. Brains class uses Behavior chain for the robot control.

Usage

Before starting the robot's program it is recommended to turn on the camera and turn off the flash (camera I used misfired very often when flash was turned on).

At startup the program ask's the user some settings in the NXT before starting to take the pictures:

  • How many horizontal steps it should have per every vertical angle. I found out that even with my not-so wide digital camera lens; 8 pictures per horizontal rotation is plenty even indoors.
  • How many vertical steps it should have. This is much more problematic than the horizontal step. With my camera you need at least 5 (including single sky shot angle) to cover the most basic outdoor panorama. I'd say more is even better, unless the sky is totally clear in which case a single sky shot can cover most of the panorama.
  • Delay before picture's are going to be taken. This is basically for the camera-man to run in/from the picture before the robot starts shooting.

After giving to robot the settings for a picture and waiting for the timer to go off; the robot initializes the vertical angle to point the floor. After then the robot does a full horizontal sweep taking pictures after which it raises the vertical angle to the next desired level. Finally after all the full 360 degree rotations around, the robot takes 1 final picture from the sky straight up.

Video of the robot in action:

After the pictures have been taken you have to transfer the images from the camera to the computer in order to start the panoramic creation. The program, Hugin is very easy to use; you simply import the pictures and it calculates automatically similarities in the pictures and creates a preview of the panoramic image. User can then do some manual finishing touches to make the image absolutely perfect, but overall it is a very fast process.

If you want the ground to be in the panoramic picture, you have to take that picture manually from underneath but I didn't bother with it during my testing.

Testing and sample pictures

After some initial problems with the horizontal gears I succeeded in making my first panoramic pictures, which turned out just perfect for my taste. Hugin software did an excellent job at joining the pictures in most cases.

The camera has to be placed on a level surface otherwise it will tip over. Best scenario would be to mount the camera on a pole or camera stand, but I really didn't want to go that far out on the project.

There is some shaking on the camera after each turn, which I countermeasured by adding a slight delay before each picture. I could have slowed down the camera rotation, but I just liked when it responded more firmly.

Some sample panoramic pictures:

Final Words

If I were to build this robot all over again, I would probably try to make the camera holding rack much more light-weight as this seems to be very critical part of the assembly. The location of the shutter motor is also bad as it adds some instability to the vertical rotation, it should be much closer to the center of mass (=on the side of the camera).

I think it would also be nice to mount the system on a pole of some sort to get images higher from the ground.

It would also be much better to use a wide lens camera if you have access to one.