» SENG 440: Lecture 23

SENG 440: Lecture 23 – CameraX

May 27, 2019 by Chris Johnson. Filed under lectures, semester1-2019, seng440.

Dear students,

Today we will create an app called Two-Face that allows the user to take a split image on the front-facing camera. The left half and right half are taken at separate times. We will use the new CameraX API that was just announced at Google I/O.

Before I forget, here’s your final TODO:

Write down on a quarter sheet of paper things about the class that you think I should keep doing and things I should change. I welcome constructive feedback.

Next lecture we will wrap up the course.

Two-Face

Following are the exercises assigned last time. We will use and discuss the solutions that you’ve submitted as we assemble our app, but I include my solutions here for reference.

Write method createPreviewUseCase that accepts parameters for the screen resolution as a Size and the screen aspect ratio as a Rational. Return a Preview for the front-facing camera, a target resolution that matches the screen resolution, a target aspect ratio that matches the screen aspect ratio, and a rotation that matches the orientation of previewView. Add a OnPreviewOutputUpdateListener to the use case that does three things:

Link the SurfaceTexture of previewView to the preview output.
Trigger updatePreviewTransform to set up the transformation of the preview. Pass the output’s texture size.

Possible solution.

private fun createPreviewUseCase(screenSize: Size, screenAspectRatio: Rational): Preview {
  val config = PreviewConfig.Builder().run {
    setLensFacing(CameraX.LensFacing.FRONT)
    setTargetResolution(screenSize)
    setTargetAspectRatio(screenAspectRatio)
    setTargetRotation(previewView.display.rotation)
    build()
  }
  val previewUseCase = Preview(config)

  previewUseCase.setOnPreviewOutputUpdateListener { previewOutput ->
    previewView.surfaceTexture = previewOutput.surfaceTexture
    updatePreviewTransform(previewOutput.textureSize)
  }

  return previewUseCase
}

Write method initializeUseCases that uses CameraX to bind preview and capture use cases. Compute the screen size and aspect ratio of previewView using the dimensions yielded by preview.display.getRealMetrics. Possible solution.

private fun initializeUseCases() {
  val metrics = DisplayMetrics().also {
    previewView.display.getRealMetrics(it)
  }
  val screenSize = Size(metrics.widthPixels, metrics.heightPixels)
  val screenAspectRatio = Rational(metrics.widthPixels, metrics.heightPixels)

  val previewUseCase = createPreviewUseCase(screenSize, screenAspectRatio)
  captureUseCase = createCaptureUseCase(screenSize, screenAspectRatio)

  CameraX.bindToLifecycle(this, previewUseCase, captureUseCase)
}

Write method createCaptureUseCase that accepts parameters for the screen resolution as a Size and the screen aspect ratio as a Rational. Return an ImageCapture for the front-facing camera, a target aspect ratio that matches the screen aspect ratio, and a rotation that matches the orientation of previewView. Possible solution.

private fun createCaptureUseCase(screenSize: Size, screenAspectRatio: Rational): ImageCapture {
  val config = ImageCaptureConfig.Builder().run {
    setLensFacing(CameraX.LensFacing.FRONT)
    setTargetAspectRatio(screenAspectRatio)
    setTargetRotation(previewView.display.rotation)
    build()
  }
  return ImageCapture(config)
}

Write method viewToRotation that accepts a View parameter. Examine the view’s display rotation. Return 0, 90, 180, or 270 accordingly. Possible solution.

private fun viewToRotation(view: View) = when (view.display.rotation) {
  Surface.ROTATION_0 -> 0
  Surface.ROTATION_90 -> 90
  Surface.ROTATION_180 -> 180
  Surface.ROTATION_270 -> 270
  else -> 0
}

Write method imageProxyToBitmap that accepts an ImageProxy parameter. Return the wrapped image as a Bitmap. (You’ll find more reference code dealing with Image than ImageProxy. Luckily, they have the same interface.) Possible solution.

private fun imageProxyToBitmap(image: ImageProxy): Bitmap {
  val buffer: ByteBuffer = image.planes[0].buffer
  val bytes = ByteArray(buffer.remaining())
  buffer.get(bytes)
  return BitmapFactory.decodeByteArray(bytes, 0, bytes.size)
}

Write method rotateBitmap that accepts parameters for a Bitmap and a number of degrees as an Int. Return a new version of the bitmap has been rotated about its center by the specified number of degrees. Use a version of Bitmap.createBitmap that accepts a Matrix. Possible solution.

private fun rotateBitmap(bitmap: Bitmap, degrees: Int): Bitmap {
  val xform = Matrix().apply {
    postRotate(degrees.toFloat(), bitmap.width / 2f, bitmap.height / 2f)
  }
  return Bitmap.createBitmap(bitmap, 0, 0, bitmap.width, bitmap.height, xform, false)
}

Write method meldBitmaps that accepts parameters for a left Bitmap and a right Bitmap. Return a new bitmap in which the two images appear side-by-side. Use the ARGB_8888 pixel format. Possible solution.

private fun meldBitmaps(left: Bitmap, right: Bitmap): Bitmap {
  val bitmap = Bitmap.createBitmap(left.width + right.width, left.height, Bitmap.Config.ARGB_8888)

  val leftPixels = IntArray(left.width * left.height)
  left.getPixels(leftPixels, 0, left.width, 0, 0, left.width, left.height)
  bitmap.setPixels(leftPixels, 0, left.width, 0, 0, left.width, left.height)

  val rightPixels = IntArray(left.width * left.height)
  right.getPixels(rightPixels, 0, right.width, 0, 0, right.width, right.height)
  bitmap.setPixels(rightPixels, 0, right.width, left.width, 0, right.width, right.height)

  return bitmap
}

Write method saveBitmap that accepts a Bitmap parameter. It stores the Bitmap as a JPEG in the top-level external storage directory under the name twoface.jpg. Possible solution.

private fun saveBitmap(bitmap: Bitmap) {
  val outFile = File(Environment.getExternalStorageDirectory(), "twoface.jpg")
  val outStream = FileOutputStream(outFile)
  bitmap.compress(Bitmap.CompressFormat.JPEG, 80, outStream)
  outStream.close()
}

Write method meldBitmapsAndSaveMaybe that melds the two bitmaps and saves the combined bitmap—but only if both leftBitmap and rightBitmap are valid. Possible solution.

private fun meldBitmapsAndSaveMaybe() {
  leftBitmap?.let { left ->
    rightBitmap?.let { right ->
      val bitmap = meldBitmaps(right, left)
      saveBitmap(bitmap)
    }
  }
}

Write method takeLeftImage that triggers the captureUseCase to take a picture in-memory. On a successful capture, convert the captured image to a bitmap, close the image, transform it according to the rotation, and crop it to contain only the left half of the image. Update the leftBitmap field, try melding the bitmaps and saving the result, then sync the left half of the UI. Possible solution.

private fun takeLeftImage() {
  captureUseCase.takePicture(object : ImageCapture.OnImageCapturedListener() {
    override fun onCaptureSuccess(image: ImageProxy, rotationDegrees: Int) {
      var bitmap = imageProxyToBitmap(image)
      image.close()

      bitmap = rotateBitmap(bitmap, rotationDegrees)
      bitmap = Bitmap.createBitmap(bitmap, bitmap.width / 2, 0, bitmap.width / 2, bitmap.height)

      leftBitmap = bitmap
      meldBitmapsAndSaveMaybe()
      syncLeft()
    }
  })
}

Write method takeRightImage that triggers the captureUseCase to take a picture in-memory. On a successful capture, convert the captured image to a bitmap, close the image, transform it according to the rotation, and crop it to contain only the right half of the image. Update the rightBitmap field, try melding the bitmaps and saving the result, then sync the right half of the UI. Possible solution.

private fun takeRightImage() {
  captureUseCase.takePicture(object : ImageCapture.OnImageCapturedListener() {
    override fun onCaptureSuccess(image: ImageProxy, rotationDegrees: Int) {
      var bitmap = imageProxyToBitmap(image)
      image.close()

      bitmap = rotateBitmap(bitmap, rotationDegrees)
      bitmap = Bitmap.createBitmap(bitmap, 0, 0, bitmap.width / 2, bitmap.height)

      rightBitmap = bitmap
      meldBitmapsAndSaveMaybe()
      syncRight()
    }
  })
}

See you next time!

Sincerely,

P.S. It’s time for a haiku!

Cameras these days
Seems they do it all for you
Even bunny ears