Cross-platform Image Preview and Capture with Xamarin Forms

Introduction

Recently, we had a need to be able to capture images efficiently and to process those images in Xamarin Forms. We were doing this image processing as part of a label reading view for our app onboarding process.

This image capture had to be:

  1. Efficient
  2. Low-memory
  3. Simple

As it turned out, this isn’t too hard to do in Xamarin Forms, but there was a lot to learn. This article will walk you through that process so you don’t have to find all of this information yourself.

If you just want the code and you’d rather skip all the explanations, you can find the complete solution on my GitHub.

Because we also process this image for text, I’ll be doing a follow-up article on how to do that. The code for the text processing is on my GitHub as well.

Custom Renderer (Android)

Because camera preview works very differently on iOS and Android, a custom renderer is needed to implement the different functionality.

For our purposes, we called this renderer CameraPageRenderer. It renders a full screen preview of the image, with optional informational text.

The constructor will look like this – we need quite a few handler classes to accomplish our task:

public CameraPageRenderer(Context context) : base(context) {
  cameraManager = (CameraManager)Context.GetSystemService(Context.CameraService);
  windowManager = Context.GetSystemService(Context.WindowService).JavaCast<IWindowManager>();
  StateCallback = new CameraStateCallback(this);
  SessionCallback = new CameraCaptureSessionCallback(this);
  CaptureListener = new CameraCaptureListener(this);
  CameraImageReaderListener = new CameraImageListener(this);
  OrientationEventListener = new CameraPageOrientationEventListener(this, Context, global::Android.Hardware.SensorDelay.Normal);
}

Definitions for these classes can be found throughout this article or on the GitHub repository.

First, you need to get camera permissions. In order to do this, you will need an event handler in your main activity called OnCameraAccepted.

public event EventHandler OnCameraAccepted;

Then you need to override OnRequestPermissionsResult in your main activity and trigger the event:

public override void OnRequestPermissionsResult(int requestCode, string[] permissions, Permission[] grantResults){
  foreach(permission in permissions){
    if(permission == Manifest.Permission.Camera) OnCameraAccepted(this, null);
  }
}

In your custom renderer for Android, make sure to subscribe to this event before checking permissions. If you have permissions, simply call your StartCamera code immediately. In our case, we’ll be doing this in the OnSurfaceTextureAvailable method:

public void OnSurfaceTextureAvailable(SurfaceTexture surface, int width, int height) {
  Surface = surface; // store our surface in our renderer

  // if the camera permission has to be accepted, then
  // the camera will be started when that happens.
  (CurrentContext as MainActivity).OnCameraAccepted += StartCamera;

  if (ContextCompat.CheckSelfPermission(CurrentContext, Manifest.Permission.Camera) != Permission.Granted) {
    ActivityCompat.RequestPermissions(CurrentContext, new string[] { Manifest.Permission.Camera }, 1);
  } else {
    StartCamera();
  }
}

Our StartCamera method should look something like this. Error handling and particulars of our application have been left out, and the constant LabelReaderConstants.MinimumUsefulImageWidthPixels should be replaced with your own:

public void StartCamera(object sender = null, EventArgs args = null) {
  string cameraId =
    GetCameraIdForOrientation(LensFacing.Back) ??
    GetCameraIdForOrientation(LensFacing.Front) ??
    GetCameraIdForOrientation(LensFacing.External);

  CameraCharacteristics characteristics = cameraManager.GetCameraCharacteristics(cameraId);
  sensorOrientation = (int)characteristics.Get(CameraCharacteristics.SensorOrientation); // store the orientation for later use

  SetupPreviewMatrix();

  // get the best size based on some minimum width for processing
  var map = (StreamConfigurationMap)characteristics.Get(CameraCharacteristics.ScalerStreamConfigurationMap);
  global::Android.Util.Size[] outputSizes = map.GetOutputSizes((int)ImageFormatType.Jpeg);
  IEnumerable<global::Android.Util.Size> bigSizes = outputSizes.Where(size => size.Width >= LabelReaderConstants.MinimumUsefulImageWidthPixels);
  if (!bigSizes.Any()) {
    bestWidth = outputSizes.Max(size => size.Width);
  } else { // use the biggest if none fit our goal width
    bestWidth = bigSizes.Min(size => size.Width);
  }

  global::Android.Util.Size bestSize = outputSizes.First(size => size.Width == bestWidth);

  // set our reader, add a listener for new images
  Reader = ImageReader.NewInstance(bestSize.Width, bestSize.Height, ImageFormatType.Jpeg, 2);
  Reader.SetOnImageAvailableListener(CameraImageReaderListener, null);
  // finally, open the camera
  cameraManager.OpenCamera(cameraId, StateCallback, null);
}

Here are the two methods referenced in the code above:

private string GetCameraIdForOrientation(LensFacing facingToMatch) {
  CameraCharacteristics characteristics = null;
  return cameraManager.GetCameraIdList().FirstOrDefault(id => {
    characteristics = cameraManager.GetCameraCharacteristics(id);
    int lensFacing = (int)characteristics.Get(CameraCharacteristics.LensFacing);
    return lensFacing == (int)facingToMatch;
  });
}

public void SetupPreviewMatrix() {
  float landscapeScreenRotation = 0.0f;
  if(windowManager.DefaultDisplay.Rotation == SurfaceOrientation.Rotation270) {
    landscapeScreenRotation = 180.0f;
  }

  float width = mainLayout.Width;
  float height = mainLayout.Height;

  Matrix matrix = new Matrix();
  matrix.PostRotate(360.0f - landscapeScreenRotation - sensorOrientation, width / 2.0f, height / 2.0f);
  if (sensorOrientation != 180) {
    matrix.PostScale(width / height, height / width, width / 2.0f, height / 2.0f);
  }
  LiveView.SetTransform(matrix);
}

SetupPreviewMatrix applies transforms to the image preview layer that will ensure it is oriented correctly for the user that is viewing it. This code only handles landscape as that’s all that was required for our project.

Opening the camera will trigger the state callback, which is where we’ll start the session and set our capture options for the preview:

public class CameraStateCallback : CameraDevice.StateCallback { 
  private readonly CameraPageRenderer _renderer;
  public CameraStateCallback(CameraPageRenderer renderer) {
    _renderer = renderer;
  }
  public override void OnOpened(CameraDevice camera) {
    // request a preview capture of the camera, and notify the session
    // that we will be rendering to the image reader, as well as the preview surface.
    _renderer.Camera = camera; // set our camera
    var surface = new Surface(_renderer.Surface); // use our stored surface (texture) to render preview
    _renderer.Builder = camera.CreateCaptureRequest(CameraTemplate.Preview);
    // auto focus the camera
    _renderer.Builder.Set(CaptureRequest.ControlAfMode, (int)ControlAFMode.ContinuousPicture);
    _renderer.Builder.Set(CaptureRequest.ControlAfTrigger, (int)ControlAFTrigger.Start);
    _renderer.Builder.AddTarget(surface);
    // start session targeting our image reader and the texture surface
    camera.CreateCaptureSession(new List<Surface> { surface, _renderer.Reader.Surface }, _renderer.SessionCallback, null);
  }
}

Once the camera is opened it will call our session callback object, which looks like this. The constant LabelReaderConstants.ImageCaptureBeginDelayMilliseconds should be replaced with your own:

public class CameraCaptureSessionCallback : CameraCaptureSession.StateCallback {
  private readonly CameraPageRenderer _renderer;
  public CameraCaptureSessionCallback(CameraPageRenderer renderer) {
    _renderer = renderer;
  }
  public override void OnConfigured(CameraCaptureSession session) {
    // set a repeating request for a live preview of the camera
    _renderer.Session = session;
    CaptureRequest request = _renderer.Builder.Build();
    _renderer.Request = request;
    session.SetRepeatingRequest(request, _renderer.CaptureListener, null);
    _renderer.CaptureImage(); // capture single image for processing
  }
}

You’ll notice you need to set a capture listener for the preview images, which you can create as an empty class like this:

public class CameraCaptureListener : CameraCaptureSession.CaptureCallback {
}

The CaptureImage method called after the session is created should look like this:

public void CaptureImage() {
  CaptureRequest.Builder builder = Camera.CreateCaptureRequest(CameraTemplate.StillCapture);
  builder.AddTarget(Reader.Surface);
  Session.Capture(builder.Build(), CaptureListener, null);
}

It simply builds a new request for a single still capture image, targeting the image reader surface. The image reader image ready listener will be called because we set it up in the StartCamera method.

Here is our image available listener for the image reader. Replace LabelReaderConstants.ImageCaptureDelayMilliseconds with a constant of your own:

public class CameraImageListener : Java.Lang.Object, ImageReader.IOnImageAvailableListener {
  private readonly CameraPageRenderer _renderer;
  public CameraImageListener(CameraPageRenderer renderer) {
    _renderer = renderer;
  }
  public void OnImageAvailable(ImageReader reader) {
    if (_renderer.CancellationToken.IsCancellationRequested) { return; }
    // get the byte array data from the first plane
    // of the image. This is sufficient for a JPEG
    // image
    Image image = reader.AcquireLatestImage();
    if (image != null) {
      Image.Plane[] planes = image.GetPlanes();
      ByteBuffer buffer = planes[0].Buffer;
      byte[] bytes = new byte[buffer.Capacity()];
      buffer.Get(bytes);
      // close the image so we can handle another image later
      image.Close();
      (_renderer.Element as LabelReader)?.ProcessPhoto(bytes);
      _renderer.CurrentContext.RunOnUiThread(async () => {
        try {
          await Task.Delay(LabelReaderConstants.ImageCaptureDelayMilliseconds, _renderer.CancellationToken);
        } catch (TaskCanceledException) {
          return;
        }
        _renderer.CaptureImage();
      });
    }
  }
}

It processes the image through our view model, capturing another image after a specified delay. Keep in mind that the CaptureImage call has to be on the main thread for an image capture event to be received.

ViewModel/View

You’ll notice we reference a LabelReader class in the code above, that handles the actual image processing. This is our view class, which we’ll explain below.

The LabelReader view is very simple. It’s just an ContentPage view that will be rendered into by our custom renderer. As such I haven’t included it here.

The view model that we bind to it is a little more interesting. We will have two views/viewmodels. The outer “parent” view is called LabelReaderPage and the inner custom renderer view is the empty once mentioned above.

The LabelReader view code-behind binds a take photo command, and exposes a public method to call it:

public partial class LabelReader : ContentPage
{
  public LabelReader ()
  {
    InitializeComponent ();
  }

  public static readonly BindableProperty TakePhotoCommandProperty =
    BindableProperty.Create(propertyName: nameof(TakePhotoCommand),
      returnType: typeof(ICommand),
      declaringType: typeof(LabelReaderPage));

  public void ProcessPhoto(object image) {
    TakePhotoCommand.Execute(image);
  }

  public void Cancel() {

  }

  /// <summary>
  /// The command for processing photo data.
  /// </summary>
  public ICommand TakePhotoCommand {
    get => (ICommand)GetValue(TakePhotoCommandProperty);
    set => SetValue(TakePhotoCommandProperty, value);
  }
}

The label reader page view simple binds this property:

<?xml version="1.0" encoding="UTF-8"?>
<views:LabelReader
  xmlns="http://xamarin.com/schemas/2014/forms"
  xmlns:x="http://schemas.microsoft.com/winfx/2009/xaml"
  xmlns:views="clr-namespace:xxx.Views;assembly=xxx"
  x:Class="xxx.Views.LabelReaderPage"
  TakePhotoCommand="{ Binding TakePhoto }">
</views:LabelReader>

This is the viewmodel that we wire to it:

public class LabelReaderPageViewModel  {
  private BufferBlock<object> ImageQueue;
  private CancellationTokenSource CancellationTokenSource;
  private CancellationToken CancellationToken;

  private Task BackgroundOperation;

  public LabelReaderPageViewModel(INavigationService navigationService) : base(navigationService) {
    CancellationTokenSource = new CancellationTokenSource();
    CancellationToken = CancellationTokenSource.Token;
    ImageQueue = new BufferBlock<object>(new DataflowBlockOptions {
      BoundedCapacity = 1
    });
    BackgroundOperation = Task.Run(() => ProcessImageAsync(CancellationToken));
  }

  private void StopBackgroundOperations() {
    CancellationTokenSource.Cancel();
  }

  private async void ProcessImageAsync(CancellationToken cancellationToken) {
    while (!cancellationToken.IsCancellationRequested) {
      object image = await ImageQueue.ReceiveAsync(cancellationToken);
      // do your image processing here
    }
  }

  /// <summary>
  /// A command that is executed when a photo is taken.
  /// </summary>
  public ICommand TakePhoto => new Command(async (object image) => {
    if (CancellationToken.IsCancellationRequested) { return; }
    // receive any pending image(s), so that our background task will get the latest image
    // when it completes processing on the previous image
    IList<object> queuedData;
    ImageQueue.TryReceiveAll(out queuedData);
    queuedData = null;
    // force GC collect our unused byte arrays
    // so we don't overflow adding another
    GC.Collect();
    GC.WaitForPendingFinalizers();
    await ImageQueue.SendAsync(image, CancellationToken);
  });
}

We use a BufferBlock with a max capacity of 1, so that if we receive more images than we can process, it won’t take up too much memory. forcing garbage collection when we receive an image is not always necessary, but it helps ensure that we don’t get memory overflow issues. We TryReceiveAll on the buffer block to clear it of any previous images we haven’t finished processing, and then we send the new image for processing in our background task, where we await the new value.

Custom Renderer (iOS)

This code in iOS is much simpler. Not in the least because the “surface” for the image preview handles rotation almost natively in their API.

public class CameraPageRenderer : PageRenderer, IAVCaptureVideoDataOutputSampleBufferDelegate {
  /// <summary>
  /// The session we have opened with the camera.
  /// </summary>
  AVCaptureSession captureSession;
  /// <summary>
  /// The camera input in our session.
  /// </summary>
  AVCaptureDeviceInput captureDeviceInput;
  /// <summary>
  /// The output class for frames from our camera session.
  /// </summary>
  AVCaptureVideoDataOutput videoDataOutput;
  /// <summary>
  /// The layer containing the video preview for still image capture
  /// </summary>
  AVCaptureVideoPreviewLayer videoPreviewLayer;
  /// <summary>
  /// The cancellation token source for canceling tasks run in the background
  /// </summary>
  CancellationTokenSource cancellationTokenSource;
  /// <summary>
  /// The cancellation token for canceling tasks run in the background
  /// </summary>
  CancellationToken cancellationToken;

  public CameraPageRenderer() : base() {

  }

  public override UIInterfaceOrientationMask GetSupportedInterfaceOrientations() {
    return UIInterfaceOrientationMask.Landscape;
  }

  protected override void OnElementChanged(VisualElementChangedEventArgs e) {
    base.OnElementChanged(e);
    SetupUserInterface();
    SetupEventHandlers();
  }

  public override void WillAnimateRotation(UIInterfaceOrientation toInterfaceOrientation, double duration) {
    base.WillAnimateRotation(toInterfaceOrientation, duration);
    videoPreviewLayer.Connection.VideoOrientation = GetCaptureOrientation(toInterfaceOrientation);
  }

  public override async void ViewDidLoad() {
    cancellationTokenSource = new CancellationTokenSource();
    cancellationToken = cancellationTokenSource.Token;
    base.ViewDidLoad();
    await AuthorizeCamera();
    SetupLiveCameraStream();
  }

  /// <summary>
  /// Gets authorization to access the camera.
  /// </summary>
  /// <returns></returns>
  async Task AuthorizeCamera() {
    var authStatus = AVCaptureDevice.GetAuthorizationStatus(AVMediaType.Video);
    if (authStatus != AVAuthorizationStatus.Authorized) {
      await AVCaptureDevice.RequestAccessForMediaTypeAsync(AVMediaType.Video);
    }
  }

  /// <summary>
  /// Gets a useable camera for the orientation we require.
  /// </summary>
  /// <param name="orientation"></param>
  /// <returns></returns>
  public AVCaptureDevice GetCameraForOrientation(AVCaptureDevicePosition orientation) {
    var devices = AVCaptureDevice.DevicesWithMediaType(AVMediaType.Video);
    foreach (var device in devices) {
      if (device.Position == orientation) {
        return device;
      }
    }
    return null;
  }

  /// <summary>
  /// Gets the orientation to capture the live preview image at
  /// based on the screen orientation. Always the nearest
  /// landscape mode.
  /// </summary>
  /// <returns></returns>
  private AVCaptureVideoOrientation GetCaptureOrientation(UIInterfaceOrientation orientation) {
  switch (orientation) {
    case UIInterfaceOrientation.LandscapeLeft:
      return AVCaptureVideoOrientation.LandscapeLeft;
    case UIInterfaceOrientation.LandscapeRight:
      return AVCaptureVideoOrientation.LandscapeRight;
    case UIInterfaceOrientation.Portrait:
      return AVCaptureVideoOrientation.LandscapeLeft;
    case UIInterfaceOrientation.PortraitUpsideDown:
      return AVCaptureVideoOrientation.LandscapeRight;
    default:
      return AVCaptureVideoOrientation.LandscapeLeft;
    }
  }

  /// <summary>
  /// Starts a session with the camera, and creates the classes
  /// needed to view a video preview, and capture a still image.
  /// </summary>
  public void SetupLiveCameraStream() {
    captureSession = new AVCaptureSession() {
      SessionPreset = new NSString(AVCaptureSession.PresetHigh)
    };
    videoPreviewLayer = new AVCaptureVideoPreviewLayer(captureSession) {
      Frame = View.Frame,
      Orientation = GetCaptureOrientation(UIApplication.SharedApplication.StatusBarOrientation)
    };
    View.Layer.AddSublayer(videoPreviewLayer);

    AVCaptureDevice captureDevice =
      GetCameraForOrientation(AVCaptureDevicePosition.Back) ??
      GetCameraForOrientation(AVCaptureDevicePosition.Front) ??
      GetCameraForOrientation(AVCaptureDevicePosition.Unspecified);

    captureDeviceInput = AVCaptureDeviceInput.FromDevice(captureDevice);
    captureSession.AddInput(captureDeviceInput);

    videoDataOutput = new AVCaptureVideoDataOutput();

    videoDataOutput.SetSampleBufferDelegateQueue(this, new CoreFoundation.DispatchQueue("frameQueue"));

    captureSession.AddOutput(videoDataOutput);
    captureSession.StartRunning();

    // set last processed time to now so the handler for video frames will wait an appropriate length of time
    // before processing images.
    lastImageProcessedTime = DateTime.Now;
  }

  /// <summary>
  /// Create the UI elements for the user interface.
  /// </summary>
  void SetupUserInterface() {
    // ui label with instructions is centered at the top.
    // to get it to appear at the top, the height must be adjusted to fit.
    // to accomplish this, I call SizeToFit, then set the frame to have
    // the same width as the screen, while preserving the height.
    UILabel takePhotoLabel = new UILabel();
    takePhotoLabel.Text = LabelReaderConstants.PhotoCaptureInstructions;
    int labelMargin = LabelReaderConstants.PhotoCaptureInstructionsMargin;
    takePhotoLabel.Frame = new CoreGraphics.CGRect(labelMargin, labelMargin, View.Frame.Width - labelMargin, View.Frame.Height - labelMargin);
    takePhotoLabel.BackgroundColor = ColorExtensions.ToUIColor(Color.Transparent);
    takePhotoLabel.TextColor = ColorExtensions.ToUIColor(Color.White);
    takePhotoLabel.TextAlignment = UITextAlignment.Center;
    takePhotoLabel.Lines = 0;
    takePhotoLabel.SizeToFit();
    takePhotoLabel.Frame = new CoreGraphics.CGRect(labelMargin, labelMargin, View.Frame.Width - labelMargin, takePhotoLabel.Frame.Height);

    View.AddSubview(takePhotoLabel);
  }

  /// <summary>
  /// Sets up event handlers for UI elements.
  /// </summary>
  void SetupEventHandlers() {

  }

  private bool imageProcessingStarted = false;
  private DateTime lastImageProcessedTime = DateTime.Now;

  [Export("captureOutput:didOutputSampleBuffer:fromConnection:")]
  public void DidOutputSampleBuffer(AVCaptureOutput captureOutput, CMSampleBuffer sampleBuffer, AVCaptureConnection connection) {
    if (!imageProcessingStarted) {
      if ((DateTime.Now - lastImageProcessedTime).TotalMilliseconds < LabelReaderConstants.ImageCaptureBeginDelayMilliseconds) { return; }
      imageProcessingStarted = true;
    }
    if((DateTime.Now - lastImageProcessedTime).TotalMilliseconds < LabelReaderConstants.ImageCaptureDelayMilliseconds) { return; }
    lastImageProcessedTime = DateTime.Now;
    (Element as LabelReader).ProcessPhoto(sampleBuffer);
  }

  public override void ViewDidUnload() {
    base.ViewDidUnload();
    cancellationTokenSource.TryCancelAndDispose();
    captureDeviceInput.TryDispose();
    videoDataOutput.TryDispose();
    captureSession.StopRunning();
    captureSession.TryDispose();
  }
}

Much of this code works very similarly to the Android code for image preview and capture. With iOS, though, a few things are simpler.

For one, we can set the orientation easily as part of their API for video capture. And due to the high quality of the video capture, we can simply process the frames that the video preview is outputting. We also do not have to delay an image capture task, but rather can simply check the last processed photo time when we receive a new frame.

If you’re wondering why we didn’t use higher quality still image captures for iOS, that’s because iOS produces a shutter sound that can be really annoying when you’re taking pictures at a constant rate, and there’s no easy way to turn this sound off.

Conclusions

Image capture with image preview cross-platform is relatively easy to accomplish with Xamarin Forms. It wasn’t simple, although the Android code is simpler if you use the older camera API.

There seems to be a need in Xamarin Forms for an image preview control that is cross-platform, and a cross-platform mechanism for capturing high-quality images from this preview.

The full code is posted on my GitHub for anyone to view and use, and is significantly simpler than the examples that it was created from. Hopefully this serves as a useful resource for those looking for this functionality in their Xamarin apps.

If there’s anything you’d like explained in more detail, please send me an email to let me know (I’ve disabled comments due to spam).

The Horror That Is React Native

Introduction

This is my story and experience with React Native development. It is meant to be a bit humorous and maybe a bit of a laying out of grievances as I come to the end of a very long and difficult development cycle for a complex and painful app. As such, there may be some things that are a bit exaggerated, and I’m only speaking for myself. So here goes 🙂

A close friend of mine came to me with an app idea recently which I finished over the course of the Summer. The idea was decent but not groundbreaking, and I didn’t want to spend too much time on it. With that in mind, I knew I would need something that could bring my web experience into the mobile arena and speed up development. The first such development kit that I was referred to was React Native.

I gave it a quick once over and it appeared to be exactly what I was looking for. It has a quick start that only would take a few lines of code. It had libraries for routing, and if you used Expo, you would get all kinds of goodies like an image picker, natively supported across multiple platforms. Little did I know the deep dark secrets lurking unseen.

Humble Beginnings

To get started, you’re presented with a couple of options. You can either do the quick start, or you can “build projects with native code”. You’d be forgiven for completely missing the second option as I did. And anyways, I chose this library because I don’t want to work with native code, right???

Running their commands as specified seems to do the trick no problem, and off you are.

Getting Real

Well this is all great, but as you start to put together an app, you notice something glaringly absent… any kind of navigation. Obviously, you don’t want to manage this yourself, and you start on your quest for tools.

This is when you start to learn that in React Native, there is an Expo version, a React version, and an open source version of nearly every solution you can encounter.

I personally used the react-navigation package, and I found their example with the stack navigator to work for my purposes. You will find later on that there is no easy way to reset the stack (for instance, when you don’t want someone to navigate back to the login screen). To fix this issue, you’ll have to use StackActions and NavigationActions to set the index back to zero. You’re welcome.

ES6 Confusion

If you’re not used to writing in ES6 JavaScript, then there’a quite a bit to learn. You will definitely learn to appreciate it in the end, but at the beginning it feels like quite the hassle.

For instance, let and const. These are not words you’d be used to seeing in JavaScript, but they are common in React Native. Another example is classes, although these are generally easy to grasp and understand by looking at them.

Classes have to be imported differently depending on whether they are declared default or not, and importing something more than once will give you a meaningless error when you build the code that can take forever to track down.

Promises use an await/async syntax similar to C#.

You can assign the property name and value of an object simultaneously by using a local variable name like this:

let a = 5;
let obj = { a }; // obj has property 'a' with value 5

Oh and when you import JS libraries that are not ES6… well the rules are weird, you’ll want to keep Google handy.

Release Horror

This is more of a small aside, as you’re unlikely to release directly from Expo anyways, but your Expo release builds will be queued up on a remote server, and will generally finish in about 15 minutes if you go the Expo quick start route. This alone made me reconsider the decision to use Expo, as the only useful thing the library gave me was an image picker.

This also requires creating an account with Expo, and if you release from these builds and later detach, you will have to contact Google support to get your publishing certificates updated. Your Expo code is always public, although you can keep it from being searchable. Would be nice if React told us all this up front wouldn’t it?

Detaching Horror

So now you have many months of working hard to put together your app, but it debugs from your PC and all you have to do is make sure your PC and phone are on the same WiFi to test your app. This all seems wonderful and great, and you’re deeply satisfied with your good decision.

However, much like relationships, React Native has hidden her true nature from you until it’s much too late, and it is on the tail end of this journey that you discover her dark secrets, for it is now time to detach.

Detaching is the process of removing your app from the Expo environment dependency, and generating native apps that run your JS code.

Detaching is quite a process, and I had it go wrong a few times before I was able to detach successfully. If you used Expo as in the quick start, then you will need to detach using the Expo documentation and not the React Native documentation, as I figured out rather quickly.

Detaching is where I faced the brunt of my issues.

First, the Google Play libraries did not match the versions given in the imported libraries, which caused all kinds of hard to track down bugs. Next, the builds only worked if I had Expo running, but the build couldn’t complete with Expo started since they accessed the same files, which made starting the JS debugger a chore. Lastly, the generated code for Android was wildly out of date and nearly every library had to be upgraded, gradle had to be upgraded, the build tools had to be upgraded, and the syntax for the gradle files had to be changed.

Everything seemed to be going much better at this point, and I figured my hard work had paid off and I could go back to an easy development cycle. Well, I got some more surprises when I tried to use the payments library I had integrated (react-native-payments).

As it turns out, the main activity code wasn’t handling Intents correctly, and it had to be updated to contain an override that looks like this:


@Override
public void onActivityResult(int requestCode, int resultCode, Intent data) {
  ((ReactInstanceManager)(Object)mReactInstanceManager.get()).onActivityResult(this, requestCode, resultCode, data );
}

So much for not writing any native code.

Now the payment module was working as expected, and I turned my eyes to the next task: push notifications. I had been originally handling this through Expo, but as it turns out that’s a complete waste of time, as you really need to use Firebase to get notifications in a production application once you detach.

The library I had moved into its place was react-native-notifications. It seemed to do everything I wanted, even though the documentation was a bit incomplete and confusing, much like react-native-payments.

At first, everything was good, I appeared to be getting the notification tokens and I happily pushed out the changes. This is when it all went wrong.

I noticed that my users weren’t receiving a single notification, even though Firebase was receiving them successfully (Oh, and Google’s terrible push notification documentation is another story).

I couldn’t for the life of me figure out why this was, not in the least because including the notifications library caused my builds to stop working with Expo, so I could no longer debug the application. I had to test full releases only. React Native was also not pushing any of my console.log messages into the android logs, so I could get no hints that way.

This is when I stumbled upon an error in the logs. The notifications library seemed to be expecting my application to be of type ReactApplication but it was instead of type ExpoApplication. Once again, Expo was breaking React through undocumented breaking issues that were incompatible with React. Of course this is React’s fault as well: they shouldn’t recommend using a library that isn’t compatible with the rest of their documentation.

I managed to find an issue on GitHub from someone else who had gone through similar horrors, and happily copied their code. But, it still didn’t work!

As I investigated the issue I found myself deep in the bowels of the React Native code, implementing obscure interfaces and overriding methods whose implementations did not fit well into the Expo model of doing things.

The final code looked something like this, and also required implementing a getReactInstanceManager() method in the main activity:

package host.exp.exponent;


import android.app.Activity;
import android.util.Log;

import com.facebook.react.ReactPackage;

import java.util.Arrays;
import java.util.List;

import expolib_v1.okhttp3.OkHttpClient;
import host.exp.expoview.Exponent;

// Needed for `react-native link`
// import com.facebook.react.ReactApplication;
import com.dieam.reactnativepushnotification.ReactNativePushNotificationPackage;
import com.reactnativepayments.ReactNativePaymentsPackage;
import com.facebook.react.ReactApplication;
import com.facebook.react.ReactNativeHost;
import com.facebook.react.ReactInstanceManager;

public class MainApplication extends ExpoApplication implements ReactApplication {

  @Override
  public boolean isDebug() {
    return BuildConfig.DEBUG;
  }

  @Override
  public ReactNativeHost getReactNativeHost() {
    return mReactNativeHost;
  }

  // Needed for `react-native link`
  public List<ReactPackage> getPackages() {
    return Arrays.<ReactPackage>asList(
        // Add your own packages here!
        // TODO: add native modules!

        // Needed for `react-native link`
        // new MainReactPackage(),
        new ReactNativePushNotificationPackage(),
        new ReactNativePaymentsPackage()
    );
  }

  private final ReactNativeHost mReactNativeHost = new ReactNativeHost(this) {

    @Override
    public ReactInstanceManager getReactInstanceManager(){
      Log.d("ReactNativeHost", "Fetching manager from Exponent (override)");
      Activity activity = Exponent.getInstance().getCurrentActivity();
      if(activity instanceof MainActivity){
        return ((MainActivity)activity).getReactInstanceManager();
      }
      Log.d("ReactNativeHost", "Activity was not main activity");
      return null;
    }

    @Override
    public boolean getUseDeveloperSupport() {
      return BuildConfig.DEBUG;
    }

    @Override
    protected List<ReactPackage> getPackages() {
      return Arrays.<ReactPackage>asList(
        new ReactNativePushNotificationPackage(),
        new ReactNativePaymentsPackage()
      );
    }
  };

  @Override
  public String gcmSenderId() {
    return getString(R.string.gcm_defaultSenderId);
  }

  @Override
  public boolean shouldUseInternetKernel() {
    return BuildVariantConstants.USE_INTERNET_KERNEL;
  }

  public static OkHttpClient.Builder okHttpClientBuilder(OkHttpClient.Builder builder) {
    // Customize/override OkHttp client here
    return builder;
  }
}

Alas, everything is working, and on I can go, slowly building each change as a new release and pushing the full apk because React Native’s recommended SDK (Expo) is completely incompatible with their own implementations.

Tooling Issues

The above issues are not the only issues I encountered on this long and arduous journey. There were missing files (gradlew.bat, which I had to generate myself), and I had to run an Expo publish (which pushes my code to their servers) before assembling each release. In the end, nearly everything that made using it worthwhile fell to pieces, and I will be spending considerable time removing Expo and bringing my app back to pure React Native, to avoid all the pitfalls of mixing the two SDKs.

Positives

Not everything about React Native and Expo was bad. In fact overall it saved me a lot of time and effort, and will continue to do so. What was truly bad about it was the way React Native encouraged developers to use an SDK (Expo) that is clearly not well-tested with their tools.

Common Language

Having a common language is a huge positive (ES6 JavaScript) and makes life so much easier. I would hate to be writing all of my business logic for both iOS and Android. The extra efforts will be more than worth it when I release on iOS as well.

COmmon Design

That same language is used for the design/styling as well in a very clever CSS-like syntax that allows for similar, though simpler behavior. Designs are done using flex layouts, which generally provide all the behavior you would usually need. Fixed positioning doesn’t always work how you’d expect, but that’s a minor gripe.

Common Assets

There’s no need to have resources for Android and have assets for iOS. You can simply manage your assets from the same directory as your JS code. This is a huge time saver.

Quick Debugging

Up until I had detached, and ran into compatibility issues with Expo and React, debugging was extremely fast and easy. It took seconds to reload a new version and I could check the code as it ran from my browser. This was a huge convenience.

it Works

It’s alive. It does what it’s supposed to. The library itself has no glaring bugs, it’s more the integration with open source code that causes the issues.

Packages

Almost everything you could ever need is supported in some package if it is not a part of the React Native SDK. This makes development very easy, although the documentation for these libraries is not always that good.

Conclusion

Don’t use Expo. Seriously. I can’t imagine why anyone would. You are way better off writing packages for React Native and building things locally than you are shifting your builds onto some server somewhere for the privilege of having an image picker built in.

Yes this means you will have to manage an Android project, but you’re going to have to do that eventually anyways. I can see no real drawback and plenty of benefits to simply skipping the quick start, skipping Expo, and using React Native only.

If you have any comments on this article let me know. I’d be happy to go into more detail on some of the issues I faced, and track down more links for bugs and documentation that I came across.

I’ll also be revising this as time goes on to include more images of some of the issues I faced for illustration.

Bugs

These are just the bugs I could find, there were at least four or five others that came up in the course of development.

https://github.com/expo/expo/issues/2073
https://github.com/naoufal/react-native-payments/issues/111
https://github.com/zo0r/react-native-push-notification/issues/846
https://github.com/zo0r/react-native-push-notification/issues/845

Links

These are some of the links I used in the course of developing the app, which may show you some of the confusion I came across during development.

https://facebook.github.io/react-native/docs/getting-started.html
https://facebook.github.io/react-native/docs/navigation