iPhone composed video preview in portrait/landscape how-to

Abstract: I address how to preview / generate thumbnail images / export a composed movie in an appropriate video orientation (portrait or landscape). For example, previewing a movie of iPod library using AVPlayer works well with any kind of its video orientation. However, once I compos the movie, its video orientation is lost in the composed movie, so AVPlayer can not show it in an adequate orientation. I demonstrate a sample application which show a movie in right orientation using a layer instruction. You can get its source code from github : https://github.com/reinforce-lab/ObjC_Codes/tree/master/moviePortraitLandscape under the New BSD license.

Introduction

I have been developing an iPhone movie editor for a month. Video composition is somehow complicated but it is not difficult if you watch WWDC 2010 “session 407 Editing Media with AV Foundation” and read thorough its sample project “AVEditDemo” source codes.

However, you have to take care of its video orientation in movie composition by yourself. Movie object is represented as an instance of AVAsset abstract class in iOS SDK. AVAsset abstract class has a property named “CGAffineTransform preferredTransform”. All movie files recorded using the default iPhone camera application has a video frame in landscape (home button is on right hand) and its video orientation can be got from the preferredTransform property of AVAsset. For example, you record a portrait movie (home button is bottom), preferredTransform is set to rotate the video frame 90 degrees to the right clockwise. Therefore when you edit a movie, you also have to set a appropriate transformation same as the original movie has.

AVMutableComposition class which inherits AVAsset is used to a compose movie but its preferredTransform property is read-only , not writable. So you can not copy transformation of a source movie to the composed movie. Creating a new AVMutableComposition inheritance class whose preferredTransform property is writable is not easy, because AVMutableComposition class is merely a wrapper class of its internal class (AVMutableComposition class method is used to instance).

How to set video orientation of a composed movie

Therefore, I rotate video frames itself using AVMutableVideoCompositionLayerInstruction to set video orientation. First I have to determine a video orientation of a source movie as following (avPlayerView.m line #338):

// AVURLAsset *asset; UIImageOrientation orientation_  = UIImageOrientationUp; BOOL  isPortrait_  = NO;     NSArray *tracks = [asset tracksWithMediaType:AVMediaTypeVideo];   if([tracks  count] != 0) {     AVAssetTrack *videoTrack = [tracks objectAtIndex:0];     CGAffineTransform t = videoTrack.preferredTransform;     if(t.a == 0 && t.b == 1.0 && t.c == -1.0 && t.d == 0)  {orientation_= UIImageOrientationRight; isPortrait_ = YES;}     if(t.a == 0 && t.b == -1.0 && t.c == 1.0 && t.d == 0)  {orientation_ =  UIImageOrientationLeft; isPortrait_ = YES;}     if(t.a == 1.0 && t.b == 0 && t.c == 0 && t.d == 1.0)   {orientation_ =  UIImageOrientationUp;}     if(t.a == -1.0 && t.b == 0 && t.c == 0 && t.d == -1.0) {orientation_ = UIImageOrientationDown;} }

As addressed before, movie is represented as a AVURLAsset instance (line #1). Video orientation which determined in line #9-12 is stored in the preferredTransform property of a video track (line #8).

Video frame rotation is set as flowing: (avPlayerView.m line #444)

AVAssetTrack *videoTrack = [[asset tracksWithMediaType:AVMediaTypeVideo] objectAtIndex:0]; AVMutableComposition *composition = [AVMutableComposition composition]; AVMutableVideoComposition *videoComposition = [AVMutableVideoComposition videoComposition];   // get frame size. rotate the source frame rectangle if it is portrait. CGSize videoSize = videoTrack.naturalSize; if(isPortrait_) {    videoSize = CGSizeMake(videoSize.height, videoSize.width);  }     // set frame size composition.naturalSize     = videoSize; // AVPlayerはこのサイズを見ない videoComposition.renderSize = videoSize; // AVPlayerはこのサイズを見る。 videoComposition.frameDuration= CMTimeMakeWithSeconds( 1 / videoTrack.nominalFrameRate, 600);   // Trackを構築 AVMutableCompositionTrack * compositionVideoTrack = [composition                addMutableTrackWithMediaType:AVMediaTypeVideo                preferredTrackID:kCMPersistentTrackID_Invalid];    [compositionVideoTrack insertTimeRange:CMTimeRangeMake(kCMTimeZero, asset.duration)                    ofTrack:videoTrack                       atTime:kCMTimeZero error:nil];     // レイヤを設定   AVMutableVideoCompositionLayerInstruction *layerInst;   layerInst = [AVMutableVideoCompositionLayerInstruction videoCompositionLayerInstructionWithAssetTrack:videoTrack]; // ここがポイント！ 変形を指定   [layerInst setTransform:videoTrack.preferredTransform atTime:kCMTimeZero];  // Layer instructionを設定 AVMutableVideoCompositionInstruction *inst = [AVMutableVideoCompositionInstruction videoCompositionInstruction]; inst.timeRange = CMTimeRangeMake(kCMTimeZero, asset.duration); inst.layerInstructions = [NSArray arrayWithObject:layerInst]; videoComposition.instructions = [NSArray arrayWithObject:inst]; // プレビュ表示 AVPlayerItem *playerItem_ = [[[AVPlayerItem alloc] initWithAsset:composition] autorelease]; playerItem_.videoComposition = videoComposition; // AVPlayer player_  [player_ replaceCurrentItemWithPlayerItem:playerItem_];

First, I instance AVMutableComposition and VideoComposition (line #1-3), setting frame and render size (#5-12) and then store the source video track into the composition (#14-20). After that, I instance a AVMutalbeVideoCompositonLayerInstruction and set preferredTransform of the source video to it (#23-32). At last, set the composed video to preview (#34-38). These usual video composition code.

Thumbnail image generation and exporting movie

This technique is also used for thumbnail image generation and exporting movie.

AVAssetImageGenerator can be used to generate multiple movie thumbnail images asynchronously. When I set its BOOL appliesPreferredTrackTransform property YES, however, image generation fails when video orientation is not landscape home right. Therefore I set the property NO and rotate the generate thumbnail images.

AVAssetExportSession can be used to export the composed asset to a local file. You can export a movie which is exactly you see in the preview. However, if you want to keep the consistency in video orientation policy (frame is always landscape right home, and video orientation is set in the preferredTransform property), you can do it by writing a custom export class (set transform property of AVAssetWriterInput).

Conclusion

I demonstrate previewing a composed asset in an adequate orientation. Although using LayerInstruction, AVPlayer can not show the movie when AVPlayerLayer videoGravity property is AspectFill, may be AVPlayerLayer does not look into the layer instructions, so it can not determine adequate layer size. So if you want to preview in AspectFill, you have to set appropriate render size and a transform by yourself.