Monday, February 8, 2010

MonoTouch_augmented reality how to

Abstract: I demonstrate a MonoTouch augmented reality application which overlays a edge detection image on the camera view. This originate at Sudoku Grab [iTunes Link] developer Chris Greening's sample application [1]. I have ported from objective-c to C# one-to-one. Using unsafe code, real-time image processing is achieved by a fast memory accessing . Its frame rate is 3 ~ 4 fps on iPhone 3GS and 1 ~ 2 on iPhone 3G. You can get the sample code from github under the New BSD license so that you can use it freely [3].

 Augmented reality (AR) applications on iPhone are very attractive that we can interact with virtual data through a camera view. Some kind of AR applications such as overlaying a name of mountains over a camera image is not so difficult. You can get compass direction and location information using iPhone SDK and can overlay images using CameraOverlay property of UIImagePickerController.

 Other kind of AR applications by text or facial recognition is much more interactive and also more attractive, however, is slightly more difficult. To develop such AR applications we need: 1) camera image capture, 2) image processing, and 3) overlaying processed information.

Capturing camera image as a gray scale bitmap image

 I have described how to get camera image in the previous blog entry [2]. We have to convert a screen image that got via static property ScreenImage of CGImage class as a CGImage object into gray-scale bitmap image for the latter image processing, edge detection. Therefore first we call a CGBitmapContext constructor as following:

   1:  using MonoTouch.CoreGraphics;
   2:  using System.Runtime.InteropServices;
   3:   
   4:  namespace com.ReinforceLab.MonoTouch.Controls.AugmentedRealityBase {
   5:   public class RawBitmap : IDisposable {
   6:    IntPtr _buffer;
   7:    CGBitmapContext _context;
   8:   
   9:    public RawBitmap(int width, int height) {
  10:     _buffer = Marshal.AllocHGlobal(width * height);
  11:     var colorSpace = CGColorSpace.CreateDeviceGray();
  12:     _context = new CGBitmapContext(
  13:                     _buffer, width, height,
  14:                     8, width,
  15:                     colorSpace,
  16:                     CGImageAlphaInfo.None);
  17:     colorSpace.Dispose();
  18:     // lowest possible quality for speed                    
  19:     _context.InterpolationQuality = CGInterpolationQuality.None;
  20:     _context.SetAllowsAntialiasing(false);
  21:    }
  22:  }}

  

 Draw() method of CGBitmapContext draws CGImage object on its memory buffer with specified parameters, size and color space (device gray scale). We have to allocate an unmanaged memory buffer at line# 10, because we can not get a memory pointer to CGBitmapContext internal buffer via Data property (it seems to be always IntPtr.Zero when CGBitmapContext allocates a buffer by itself).


   1:  public void Draw(CGImage image) {
   2:    _context.DrawImage(new RectangleF(0, 0, _context.Width, _context.Height), image);
   3:  }

Image processing

 Before proceeding to edge detection, we have to remove overlay view from the screen captured image. The following figures are: a) a camera image, b) a camera image in a gray scale format, and c) screen image consisted of a camera image and an edge detection image (the overlay image is colored in green for purposes of illustration). 



 The gray scale image we capture (line# 3 - 5) is the figure (c), not the figure (b). Therefore a pixel value of a green colored pixel in the figure (c) is replaced with an average value of the nearest pixels (line# 6 - 21). After that pre-processing, edge detection is processed (line# 24 - 43) using unsafe code block to achieve higher frame rate. Finally we call SetNeedDisplay() at line# 46 to evoke a UIView.Draw() method.


   1:  void worker() {
   2:   // turn screen image into gray-scale raw bitmap byte-stream
   3:   using (var screenImage = CGImage.ScreenImage)
   4:       using(var rectImage = screenImage.WithImageInRect(Frame))
   5:           _buffer.Draw(rectImage);
   6:   // process the image to remove our drawing - WARNING the edge pixels of the image are not processed 
   7:   if (null != _drawnImage) {
   8:   unsafe {
   9:    var bufPtr   = _buffer.BaseAddress;
  10:    var drawnPtr = _drawnImage.BaseAddress;
  11:    var height   = _buffer.Height;
  12:    var width    = _buffer.Width;
  13:    // if we draw to this pixel replace it with the average of the surrounding pixels
  14:    for (int y = 1; y < height-1; y++) {
  15:     var scan0 = width * (y -1); var scan1 = width * y; var scan2 = width * (y +1);
  16:     for (int x = 1; x < width-1; x++) {
  17:      ++scan0; ++scan1; ++scan2;
  18:      if (drawnPtr[scan1] != 0) {
  19:          var val = (bufPtr[scan0] + bufPtr[scan2] + bufPtr[scan1 -1] + bufPtr[scan1 + 1]) / 4;
  20:          bufPtr[scan1] = (Byte)val;                                
  21:      }}}}}
  22:   
  23:   // edge detection
  24:   var path = new CGPath();
  25:   int lastX = -1000, lastY = -1000;
  26:   unsafe {                    
  27:   var bufPtr   = _buffer.BaseAddress; var height   = _buffer.Height;  var width    = _buffer.Width;
  28:   for (int y = 0; y < height - 1; y++) {
  29:    var scan0 = width * y; var scan1 = width * (y + 1);
  30:    for (int x = 0; x < width - 1; x++) {
  31:     int edge = (Math.Abs(bufPtr[scan0] - bufPtr[scan0 + 1]) + Math.Abs(bufPtr[scan0] - bufPtr[scan1])) / 2;
  32:     if (edge > 10) {
  33:      int dist = (int)(Math.Pow(x - lastX, 2) + Math.Pow(y - lastY, 2));
  34:      if (dist > 50) {
  35:       lastX = x;
  36:       lastY = y;
  37:      } else if (dist > 10) {
  38:       path.AddLines(new PointF[] { new PointF(lastX, lastY), new PointF(x, y) });
  39:       lastX = x;
  40:       lastY = y;
  41:      }}
  42:     ++scan0; ++scan1;
  43:    }}}
  44:   
  45:   _path = path;
  46:   SetNeedsDisplay();
  47:  }

Image overlay

 Camera image overlay is also described in the previous blog entry [2]. UIView.Draw() method shown in the following draws paths on image edges with a checkered mask image (line# 3 - 12). Coordination conversion from CoreGraphics to UIView is done at line# 16 - 17. Overlay image is preserved for the next pre-processing mentioned above at line# 23.


   1:  public override void Draw(System.Drawing.RectangleF rect) {
   2:   // we're going to draw into an image using our checkerboard mask
   3:   UIGraphics.BeginImageContext(this.Bounds.Size);
   4:   var context = UIGraphics.GetCurrentContext();
   5:   context.ClipToMask(this.Bounds, _maskImage);
   6:   // do your drawing here                
   7:   context.SetLineWidth(2);
   8:   context.SetStrokeColorWithColor(UIColor.Green.CGColor);
   9:   context.AddPath(_path);
  10:   context.StrokePath();
  11:   var imageToDraw = UIGraphics.GetImageFromCurrentImageContext();
  12:   UIGraphics.EndImageContext();
  13:   
  14:   // now do the actual drawing of the image
  15:   var drawContext = UIGraphics.GetCurrentContext();
  16:   drawContext.TranslateCTM(0f, Bounds.Size.Height);
  17:   drawContext.ScaleCTM(1.0f, -1.0f);
  18:   // very important to switch these off - we don't wnat our grid pattern to be disturbed in any way
  19:   drawContext.InterpolationQuality = CGInterpolationQuality.None;
  20:   drawContext.SetAllowsAntialiasing(false);
  21:   drawContext.DrawImage(Frame, imageToDraw.CGImage);
  22:   // stash the results of our drawing so we can remove them later
  23:   _drawnImage.Draw(imageToDraw.CGImage);
  24:   imageToDraw.Dispose();
  25:   
  26:   _path.Dispose();
  27:   _path = null;
  28:  }

Discussion

 Higher frame rate, to say higher throughput, is our concern. Besides code optimization, we should consider to utilize hardware accelerated signal processing of iPhone, such as NEON technology or GPU processing. I hope MonoTouch supports Mono.SIMD.

Acknowledgments

 This sample code is derived from  Sudoku Grab [iTunes Link] developer Chris Greening's sample application [1]. I would thank to Miguel de Icaza (http://twitter.com/migueldeicaza) and Geoff Norton (http://twitter.com/kangamono) for usefull suggestions.

References

  1. Augmented Reality on the iPhone - how to, http://cmgresearch.blogspot.com/2010/01/augmented-reality-on-iphone-how-to_01.html
  2. MonoTouch video capture in c# code, http://blog.reinforce-lab.com/2010/02/monotouchvideocapturinghowto.html
  3. Sample code on github, http://github.com/reinforce-lab/com.ReinforceLab.MonoTouch.Controls/tree/master/AugmentedRealityBase/
  4. Augmented reality - wikipedia, http://en.wikipedia.org/wiki/Augmented_reality

12 comments:

  1. Very good!
    I know that MonoTouch is compiled to native code.
    Is there any performance differ between MonoTouch and native objective-c? If no, why using Unsafe in C#?

    ReplyDelete
  2. This comment has been removed by the author.

    ReplyDelete
  3. Unsafe code is fast as twice as c# managed code is (rough estimation).
    I have not shown details here. But it is not surprising. Managed code requires array indexer to be checked whether it is within its range or not, but unsafe code does not require it.

    ReplyDelete
  4. The only difference in using MonoTouch and pure CocoaTouch is that everything you define in the first will be garbage collected.
    There is no "managed code" really as it is compiled, accessing pointers, without VMs.

    "managed code" = code that is run, managed, garbage-collected, inside a virtual machine (interpreted or JITted)
    "unsafe" = declaring pointers and accessing things outside the normal scope of what you defined (so, unsafe as "can create a pointer to kernel and crash everything")

    ReplyDelete
  5. Hi,
    can i find a download link somewhere?

    ReplyDelete
  6. Sure, you can find the source code here, https://github.com/reinforce-lab/com.ReinforceLab.MonoTouch.Controls/tree/master/AugmentedRealityBase/

    FYI:
    iOS4 provides AVCaptureSession to real-time video frame capturing. To achieve faster image processing, the blog entry GPU-accelerated video processing http://www.sunsetlakesoftware.com/2010/10/22/gpu-accelerated-video-processing-mac-and-ios would be very helpful. Using this technique, you can accelerate by the factor of 14.

    ReplyDelete
  7. This comment has been removed by the author.

    ReplyDelete
  8. I have read so many articles or reviews on the topic of the blogger
    lovers however this post is truly a pleasant piece of writing, keep it
    up.

    My homepage; how to get rid of cellulite fast and naturally

    ReplyDelete
  9. Since our meal could cause lethargy and exhaustion by containing unwanted substances, taking Aloe Vera capsules can help cleanse these far from
    our bodies. Alternatively, applying Aloe Vera Gel topically will not
    be competitive with apply yogurt, that may actually assistance
    to cure chlamydia. It is considered that Aloe Vera has been use since 16th century BCE.


    my site; aloe vera juice reviews

    ReplyDelete
  10. My coder is trying to persuade me to move to .net from
    PHP. I have always disliked the idea because of the costs.
    But he's tryiong none the less. I've been using Movable-type on a variety
    of websites for about a year and am worried about switching to another platform.
    I have heard good things about blogengine.net.
    Is there a way I can import all my wordpress posts into it?
    Any kind of help would be greatly appreciated!

    my webpage ... wealthy affiliate scam or not

    ReplyDelete
  11. Hi to all, it's truly a good for me to go to see this site, it consists of helpful Information.

    my webpage: best way to get more followers on instagram

    ReplyDelete