MonoTouch_augmented reality how to

Abstract: I demonstrate a MonoTouch augmented reality application which overlays a edge detection image on the camera view. This originate at Sudoku Grab [iTunes Link] developer Chris Greening's sample application [1]. I have ported from objective-c to C# one-to-one. Using unsafe code, real-time image processing is achieved by a fast memory accessing . Its frame rate is 3 ~ 4 fps on iPhone 3GS and 1 ~ 2 on iPhone 3G. You can get the sample code from github under the New BSD license so that you can use it freely [3].

 Augmented reality (AR) applications on iPhone are very attractive that we can interact with virtual data through a camera view. Some kind of AR applications such as overlaying a name of mountains over a camera image is not so difficult. You can get compass direction and location information using iPhone SDK and can overlay images using CameraOverlay property of UIImagePickerController.

 Other kind of AR applications by text or facial recognition is much more interactive and also more attractive, however, is slightly more difficult. To develop such AR applications we need: 1) camera image capture, 2) image processing, and 3) overlaying processed information.

Capturing camera image as a gray scale bitmap image

 I have described how to get camera image in the previous blog entry [2]. We have to convert a screen image that got via static property ScreenImage of CGImage class as a CGImage object into gray-scale bitmap image for the latter image processing, edge detection. Therefore first we call a CGBitmapContext constructor as following:

   1:  using MonoTouch.CoreGraphics;

   2:  using System.Runtime.InteropServices;

   3:   

   4:  namespace com.ReinforceLab.MonoTouch.Controls.AugmentedRealityBase {

   5:   public class RawBitmap : IDisposable {

   6:    IntPtr _buffer;

   7:    CGBitmapContext _context;

   8:   

   9:    public RawBitmap(int width, int height) {

  10:     _buffer = Marshal.AllocHGlobal(width * height);

  11:     var colorSpace = CGColorSpace.CreateDeviceGray();

  12:     _context = new CGBitmapContext(

  13:                     _buffer, width, height,

  14:                     8, width,

  15:                     colorSpace,

  16:                     CGImageAlphaInfo.None);

  17:     colorSpace.Dispose();

  18:     // lowest possible quality for speed                    

  19:     _context.InterpolationQuality = CGInterpolationQuality.None;

  20:     _context.SetAllowsAntialiasing(false);

  21:    }

  22:  }}

  

 Draw() method of CGBitmapContext draws CGImage object on its memory buffer with specified parameters, size and color space (device gray scale). We have to allocate an unmanaged memory buffer at line# 10, because we can not get a memory pointer to CGBitmapContext internal buffer via Data property (it seems to be always IntPtr.Zero when CGBitmapContext allocates a buffer by itself).

   1:  public void Draw(CGImage image) {

   2:    _context.DrawImage(new RectangleF(0, 0, _context.Width, _context.Height), image);

   3:  }

Image processing

 Before proceeding to edge detection, we have to remove overlay view from the screen captured image. The following figures are: a) a camera image, b) a camera image in a gray scale format, and c) screen image consisted of a camera image and an edge detection image (the overlay image is colored in green for purposes of illustration). 

 The gray scale image we capture (line# 3 - 5) is the figure (c), not the figure (b). Therefore a pixel value of a green colored pixel in the figure (c) is replaced with an average value of the nearest pixels (line# 6 - 21). After that pre-processing, edge detection is processed (line# 24 - 43) using unsafe code block to achieve higher frame rate. Finally we call SetNeedDisplay() at line# 46 to evoke a UIView.Draw() method.

   1:  void worker() {

   2:   // turn screen image into gray-scale raw bitmap byte-stream

   3:   using (var screenImage = CGImage.ScreenImage)

   4:       using(var rectImage = screenImage.WithImageInRect(Frame))

   5:           _buffer.Draw(rectImage);

   6:   // process the image to remove our drawing - WARNING the edge pixels of the image are not processed 

   7:   if (null != _drawnImage) {

   8:   unsafe {

   9:    var bufPtr   = _buffer.BaseAddress;

  10:    var drawnPtr = _drawnImage.BaseAddress;

  11:    var height   = _buffer.Height;

  12:    var width    = _buffer.Width;

  13:    // if we draw to this pixel replace it with the average of the surrounding pixels

  14:    for (int y = 1; y < height-1; y++) {

  15:     var scan0 = width * (y -1); var scan1 = width * y; var scan2 = width * (y +1);

  16:     for (int x = 1; x < width-1; x++) {

  17:      ++scan0; ++scan1; ++scan2;

  18:      if (drawnPtr[scan1] != 0) {

  19:          var val = (bufPtr[scan0] + bufPtr[scan2] + bufPtr[scan1 -1] + bufPtr[scan1 + 1]) / 4;

  20:          bufPtr[scan1] = (Byte)val;                                

  21:      }}}}}

  22:   

  23:   // edge detection

  24:   var path = new CGPath();

  25:   int lastX = -1000, lastY = -1000;

  26:   unsafe {                    

  27:   var bufPtr   = _buffer.BaseAddress; var height   = _buffer.Height;  var width    = _buffer.Width;

  28:   for (int y = 0; y < height - 1; y++) {

  29:    var scan0 = width * y; var scan1 = width * (y + 1);

  30:    for (int x = 0; x < width - 1; x++) {

  31:     int edge = (Math.Abs(bufPtr[scan0] - bufPtr[scan0 + 1]) + Math.Abs(bufPtr[scan0] - bufPtr[scan1])) / 2;

  32:     if (edge > 10) {

  33:      int dist = (int)(Math.Pow(x - lastX, 2) + Math.Pow(y - lastY, 2));

  34:      if (dist > 50) {

  35:       lastX = x;

  36:       lastY = y;

  37:      } else if (dist > 10) {

  38:       path.AddLines(new PointF[] { new PointF(lastX, lastY), new PointF(x, y) });

  39:       lastX = x;

  40:       lastY = y;

  41:      }}

  42:     ++scan0; ++scan1;

  43:    }}}

  44:   

  45:   _path = path;

  46:   SetNeedsDisplay();

  47:  }

Image overlay

 Camera image overlay is also described in the previous blog entry [2]. UIView.Draw() method shown in the following draws paths on image edges with a checkered mask image (line# 3 - 12). Coordination conversion from CoreGraphics to UIView is done at line# 16 - 17. Overlay image is preserved for the next pre-processing mentioned above at line# 23.

   1:  public override void Draw(System.Drawing.RectangleF rect) {

   2:   // we're going to draw into an image using our checkerboard mask

   3:   UIGraphics.BeginImageContext(this.Bounds.Size);

   4:   var context = UIGraphics.GetCurrentContext();

   5:   context.ClipToMask(this.Bounds, _maskImage);

   6:   // do your drawing here                

   7:   context.SetLineWidth(2);

   8:   context.SetStrokeColorWithColor(UIColor.Green.CGColor);

   9:   context.AddPath(_path);

  10:   context.StrokePath();

  11:   var imageToDraw = UIGraphics.GetImageFromCurrentImageContext();

  12:   UIGraphics.EndImageContext();

  13:   

  14:   // now do the actual drawing of the image

  15:   var drawContext = UIGraphics.GetCurrentContext();

  16:   drawContext.TranslateCTM(0f, Bounds.Size.Height);

  17:   drawContext.ScaleCTM(1.0f, -1.0f);

  18:   // very important to switch these off - we don't wnat our grid pattern to be disturbed in any way

  19:   drawContext.InterpolationQuality = CGInterpolationQuality.None;

  20:   drawContext.SetAllowsAntialiasing(false);

  21:   drawContext.DrawImage(Frame, imageToDraw.CGImage);

  22:   // stash the results of our drawing so we can remove them later

  23:   _drawnImage.Draw(imageToDraw.CGImage);

  24:   imageToDraw.Dispose();

  25:   

  26:   _path.Dispose();

  27:   _path = null;

  28:  }

Discussion

 Higher frame rate, to say higher throughput, is our concern. Besides code optimization, we should consider to utilize hardware accelerated signal processing of iPhone, such as NEON technology or GPU processing. I hope MonoTouch supports Mono.SIMD.

Acknowledgments

 This sample code is derived from  Sudoku Grab [iTunes Link] developer Chris Greening's sample application [1]. I would thank to Miguel de Icaza (http://twitter.com/migueldeicaza) and Geoff Norton (http://twitter.com/kangamono) for usefull suggestions.

References

1. Augmented Reality on the iPhone - how to, http://cmgresearch.blogspot.com/2010/01/augmented-reality-on-iphone-how-to_01.html
2. MonoTouch video capture in c# code, http://blog.reinforce-lab.com/2010/02/monotouchvideocapturinghowto.html
3. Sample code on github, http://github.com/reinforce-lab/com.ReinforceLab.MonoTouch.Controls/tree/master/AugmentedRealityBase/
4. Augmented reality - wikipedia, http://en.wikipedia.org/wiki/Augmented_reality