Today we wrap up our course. Let’s recount what happened this semester. In week 1 we examined Kotlin, a language that many of us hadn’t used before—myself included. It seemed like the right thing to do, and that decision was recently validated by Android chief advocate Chet Haase at Google I/O:

We’re announcing that the next big step that we’re taking is that we’re going Kotlin-first. We understand that not everybody is on Kotlin right now, but we believe that you should get there.

If you’re starting a new project, you should write it in Kotlin; code written in Kotlin often means much less code for you—less code to type, test, and maintain.

In week 2 we explored the cornerstone of interactive apps: the Activity class. In Moarse, an app to emit Morse code signals, we wired up our first buttons and lambda callbacks. In TwoTimer, an app for showing the time in two different timezones, we explored scheduling tasks with Handler and ways of making our activities aware of lifecycle events so that they don’t forget who they were before the screen rotated or the language changed.

In week 3 we saw how our apps reside in a large service-oriented ecosystem. In Connect440, we assembled our first list and used implicit intents to trigger other apps to email, call, or map a classmate. For your information, this was also the day that the projector kept flickering on me. In Kana-san, a flashcard app for learning the two Japanese phonetic alphabets, we built our first app with more than one screen, which we reached via an explicit intent.

In week 4 we had our first look at persistence. We wrote Is Kitten, a game that has the potential to model the entire world in a dichotomous key, which we wrote to internal storage. We also started our work on Lately, an app for showing the latest headlines. The headlines were fetched from the NewsAPI web service, giving us our first taste of connecting to the web. We saw that we can’t issue network requests on the main thread, so we conjured up an AsyncTask.

In week 5 we discussed a pillar of mobile platforms everywhere: the list view. Because our collections can grow large and have complicated views, we saw how lists can impose a view holder pattern separating allocation of views from the population of their content. We added a RecyclerView to Lately. Then, in order to implement a master+detail UI pattern, we explored fragments as a way to form reusable portions of user interface.

In week 6 we examined managing resources that were sensitive to a device’s screen size, orientation, and language. We put Lately to bed, thank goodness, and moved on to dealing with media. We built Rattler to let the user compose RTTTL ringtones and used MediaPlayer to play them.

In week 7 we looked at object-relational mapping tool Room to bridge between a database and Kotlin. We extended Rattler to persist a collection of songs to a local database. In our last meeting of term 1, we used the alarm and notification system to send periodic reminders to the user. We started building Backlog, an app for taking daily pictures.

Then we all went on break. I saw glow worms with my family. My youngest children got car sick. I had a crisis that caused me to restructure the lectures to be more about the code that you write.

In week 8 we acquired images from the camera and image gallery. We used FileProvider to deal with the security issues of sharing resources between apps. That rounded out our Backlog app—which I have been using when the notifications work.

In week 9, we sensed the world. We wrote Lonely Phone to cry for help when the gravity sensor says that gravity is up or down. We wrote Hideaway to pin a hidden message to a geographic location and located our device with GPS to determine if we could unlock it.

In week 10, we connected devices to one another in P2Piano. This app presented a shared piano to two users connected via Google Nearby. We also touched upon multitouch, writing Lots, an app for randomly choosing one of many possible fingers that are touching the screen.

In week 11, we explored scaling and panning the interface with gestures. In Recog, we gave the user an anagram to solve and then listened for their response using speech recogniton.

In week 12, we had our first look at the new CameraX API to write Two-Face, an app for taking split-screen images. We are probably one of the first mobile development courses in the world to have used CameraX. Because, my friends, this stuff moves fast.

Many of these lectures ended with apps in semi-broken states. This is a problem I have. My dreams continue to not fit in my allotted time. The good news is that working versions of all our apps are available on GitHub:

• Moarse
• TwoTimer
• Connect440
• Kana-san
• IsKitten
• Lately
• Rattler
• Backlog
• Lonely Phone
• Hideaway
• P2Piano
• Lots
• ZoomPan
• Recog
• Two-Face

For our final exam, you will be asked to implement a collection of tasks common to Android app development. In particular, you will be given a video of me interacting with an app, and you will reverse engineer what you see as you recreate that app. I’d like to hear from you what you think every Android developer should be comfortable doing quickly and routinely. Together we will generate a list of features that you can expect to see on the exam.

Across our two terms together, we wrote apps that fit the culture of mobility that we described in the first lecture.

• Our apps are casual, designed for informal and interruptible use.
• Our apps are paranoid about getting disrupted, so we persisted our data.
• Our apps are personal, managing our own data for daily living.
• Our apps are social, connecting us to friends and those around us.

In what you would call year 6, my social studies teacher Mr. Schmidt told us that he teaches life skills, and that social studies is only his vehicle. I suppose my vehicle is computer science, but I’m not quite sure what I teach. If I had to settle on a single big idea that motivated my design of this course it would be this: build stuff for yourself.

This sounds selfish. Really, it’s a reaction to the capitalistic tendencies of our industry. Many of us will get jobs and work on business products. But business is only one dimension of our lives. We need apps that go with us when we tramp, to entertain us at bus stops, and to bind us to family and friends. Because our devices are with us and connected to us, it’s easier on mobile platforms to build software that reaches into the small parts of our day, to affect individual humans and their emotions.

I encourage you to use the skills that you’ve acquired in this course to make things for yourself and your communities.

See you at your demos!

Sincerely,

P.S. It’s time for a haiku!

Not money, not stars
My app’s greatest achievement?
When Mom installed it

Today we will create an app called Two-Face that allows the user to take a split image on the front-facing camera. The left half and right half are taken at separate times. We will use the new CameraX API that was just announced at Google I/O.

Before I forget, here’s your final TODO:

• Write down on a quarter sheet of paper things about the class that you think I should keep doing and things I should change. I welcome constructive feedback.

Next lecture we will wrap up the course.

Two-Face

Following are the exercises assigned last time. We will use and discuss the solutions that you’ve submitted as we assemble our app, but I include my solutions here for reference.

1. Write method createPreviewUseCase that accepts parameters for the screen resolution as a Size and the screen aspect ratio as a Rational. Return a Preview for the front-facing camera, a target resolution that matches the screen resolution, a target aspect ratio that matches the screen aspect ratio, and a rotation that matches the orientation of previewView. Add a OnPreviewOutputUpdateListener to the use case that does three things:
   • Link the SurfaceTexture of previewView to the preview output.
   • Trigger updatePreviewTransform to set up the transformation of the preview. Pass the output’s texture size.

   Possible solution.

   private fun createPreviewUseCase(screenSize: Size, screenAspectRatio: Rational): Preview {
     val config = PreviewConfig.Builder().run {
       setLensFacing(CameraX.LensFacing.FRONT)
       setTargetResolution(screenSize)
       setTargetAspectRatio(screenAspectRatio)
       setTargetRotation(previewView.display.rotation)
       build()
     }
     val previewUseCase = Preview(config)
   
     previewUseCase.setOnPreviewOutputUpdateListener { previewOutput ->
       previewView.surfaceTexture = previewOutput.surfaceTexture
       updatePreviewTransform(previewOutput.textureSize)
     }
   
     return previewUseCase
   }
   

2. Write method initializeUseCases that uses CameraX to bind preview and capture use cases. Compute the screen size and aspect ratio of previewView using the dimensions yielded by preview.display.getRealMetrics. Possible solution.

   private fun initializeUseCases() {
     val metrics = DisplayMetrics().also {
       previewView.display.getRealMetrics(it)
     }
     val screenSize = Size(metrics.widthPixels, metrics.heightPixels)
     val screenAspectRatio = Rational(metrics.widthPixels, metrics.heightPixels)
   
     val previewUseCase = createPreviewUseCase(screenSize, screenAspectRatio)
     captureUseCase = createCaptureUseCase(screenSize, screenAspectRatio)
   
     CameraX.bindToLifecycle(this, previewUseCase, captureUseCase)
   }
   

3. Write method createCaptureUseCase that accepts parameters for the screen resolution as a Size and the screen aspect ratio as a Rational. Return an ImageCapture for the front-facing camera, a target aspect ratio that matches the screen aspect ratio, and a rotation that matches the orientation of previewView. Possible solution.

   private fun createCaptureUseCase(screenSize: Size, screenAspectRatio: Rational): ImageCapture {
     val config = ImageCaptureConfig.Builder().run {
       setLensFacing(CameraX.LensFacing.FRONT)
       setTargetAspectRatio(screenAspectRatio)
       setTargetRotation(previewView.display.rotation)
       build()
     }
     return ImageCapture(config)
   }
   

4. Write method viewToRotation that accepts a View parameter. Examine the view’s display rotation. Return 0, 90, 180, or 270 accordingly. Possible solution.

   private fun viewToRotation(view: View) = when (view.display.rotation) {
     Surface.ROTATION_0 -> 0
     Surface.ROTATION_90 -> 90
     Surface.ROTATION_180 -> 180
     Surface.ROTATION_270 -> 270
     else -> 0
   }
   

5. Write method imageProxyToBitmap that accepts an ImageProxy parameter. Return the wrapped image as a Bitmap. (You’ll find more reference code dealing with Image than ImageProxy. Luckily, they have the same interface.) Possible solution.

   private fun imageProxyToBitmap(image: ImageProxy): Bitmap {
     val buffer: ByteBuffer = image.planes[0].buffer
     val bytes = ByteArray(buffer.remaining())
     buffer.get(bytes)
     return BitmapFactory.decodeByteArray(bytes, 0, bytes.size)
   }
   

6. Write method rotateBitmap that accepts parameters for a Bitmap and a number of degrees as an Int. Return a new version of the bitmap has been rotated about its center by the specified number of degrees. Use a version of Bitmap.createBitmap that accepts a Matrix. Possible solution.

   private fun rotateBitmap(bitmap: Bitmap, degrees: Int): Bitmap {
     val xform = Matrix().apply {
       postRotate(degrees.toFloat(), bitmap.width / 2f, bitmap.height / 2f)
     }
     return Bitmap.createBitmap(bitmap, 0, 0, bitmap.width, bitmap.height, xform, false)
   }
   

7. Write method meldBitmaps that accepts parameters for a left Bitmap and a right Bitmap. Return a new bitmap in which the two images appear side-by-side. Use the ARGB_8888 pixel format. Possible solution.

   private fun meldBitmaps(left: Bitmap, right: Bitmap): Bitmap {
     val bitmap = Bitmap.createBitmap(left.width + right.width, left.height, Bitmap.Config.ARGB_8888)
   
     val leftPixels = IntArray(left.width * left.height)
     left.getPixels(leftPixels, 0, left.width, 0, 0, left.width, left.height)
     bitmap.setPixels(leftPixels, 0, left.width, 0, 0, left.width, left.height)
   
     val rightPixels = IntArray(left.width * left.height)
     right.getPixels(rightPixels, 0, right.width, 0, 0, right.width, right.height)
     bitmap.setPixels(rightPixels, 0, right.width, left.width, 0, right.width, right.height)
   
     return bitmap
   }
   

8. Write method saveBitmap that accepts a Bitmap parameter. It stores the Bitmap as a JPEG in the top-level external storage directory under the name twoface.jpg. Possible solution.

   private fun saveBitmap(bitmap: Bitmap) {
     val outFile = File(Environment.getExternalStorageDirectory(), "twoface.jpg")
     val outStream = FileOutputStream(outFile)
     bitmap.compress(Bitmap.CompressFormat.JPEG, 80, outStream)
     outStream.close()
   }
   

9. Write method meldBitmapsAndSaveMaybe that melds the two bitmaps and saves the combined bitmap—but only if both leftBitmap and rightBitmap are valid. Possible solution.

   private fun meldBitmapsAndSaveMaybe() {
     leftBitmap?.let { left ->
       rightBitmap?.let { right ->
         val bitmap = meldBitmaps(right, left)
         saveBitmap(bitmap)
       }
     }
   }
   

10. Write method takeLeftImage that triggers the captureUseCase to take a picture in-memory. On a successful capture, convert the captured image to a bitmap, close the image, transform it according to the rotation, and crop it to contain only the left half of the image. Update the leftBitmap field, try melding the bitmaps and saving the result, then sync the left half of the UI. Possible solution.

    private fun takeLeftImage() {
      captureUseCase.takePicture(object : ImageCapture.OnImageCapturedListener() {
        override fun onCaptureSuccess(image: ImageProxy, rotationDegrees: Int) {
          var bitmap = imageProxyToBitmap(image)
          image.close()
    
          bitmap = rotateBitmap(bitmap, rotationDegrees)
          bitmap = Bitmap.createBitmap(bitmap, bitmap.width / 2, 0, bitmap.width / 2, bitmap.height)
    
          leftBitmap = bitmap
          meldBitmapsAndSaveMaybe()
          syncLeft()
        }
      })
    }
    

11. Write method takeRightImage that triggers the captureUseCase to take a picture in-memory. On a successful capture, convert the captured image to a bitmap, close the image, transform it according to the rotation, and crop it to contain only the right half of the image. Update the rightBitmap field, try melding the bitmaps and saving the result, then sync the right half of the UI. Possible solution.

    private fun takeRightImage() {
      captureUseCase.takePicture(object : ImageCapture.OnImageCapturedListener() {
        override fun onCaptureSuccess(image: ImageProxy, rotationDegrees: Int) {
          var bitmap = imageProxyToBitmap(image)
          image.close()
    
          bitmap = rotateBitmap(bitmap, rotationDegrees)
          bitmap = Bitmap.createBitmap(bitmap, 0, 0, bitmap.width / 2, bitmap.height)
    
          rightBitmap = bitmap
          meldBitmapsAndSaveMaybe()
          syncRight()
        }
      })
    }
    

See you next time!

Sincerely,

P.S. It’s time for a haiku!

Cameras these days
Seems they do it all for you
Even bunny ears

Today we will create an app called Recog that presents anagrams for the user to unscramble. But instead of typing, the user will speak the answer. We’ll use Android’s speech recognition facilities to make this happen.

Next lecture we will explore the new CameraX API that was just announced at Google I/O 2019. See the TODO below for the assigned exercises.

Consider this extra credit opportunity:

On Friday, 24 May at 1 PM, Chris Johnson will presenting a seminar entitled Computational Making in JE445. Attend and write down a response on a quarter sheet for an extra 0.5 Pakipaki.

Recog

Following are the exercises assigned last time. We will use and discuss the solutions that you’ve submitted as we assemble our app, but I include my solutions here for reference.

1. Write extension function anagram for String. It generates a new String that is a random shuffling of the letters of the original String. Ensure that the shuffling isn’t identical to the original String. Possible solution.

   fun String.anagram(): String {
     var candidate: String
     do {
       candidate = this.toCharArray().toList().shuffled().joinToString("")
     } while (candidate == this)
     return candidate
   }
   

2. Write extension function hasSameLetters for String that accepts another String as a parameter. It returns true if the two Strings have the same letters. For example, "these".hasSameLetters("sheet") returns true. We need this method because some anagrams may have multiple correct answers, and we want to accept any of them, not just the one we have in mind. For example, reab could unscramble to either bear or bare. Both have the same pronunciation. So it is with break and brake. Can you think of others? Possible solution.

   fun String.hasSameLetters(other: String) =
     this.toCharArray().sort() == other.toCharArray().sort()
   

3. Define class RandomWordFetcher to extend AsyncTask and accept a MainActivity, a host string, and a key string as constructor parameters. In the background, fetch the JSON from the Words API endpoint at https://wordsapiv1.p.mashape.com/words/, with the query parameters random=true, frequencyMin=4, and letters=5. Send also the headers X-RapidAPI-Host and X-RapidAPI-Key with the specified strings. The resulting JSON will be formatted according to Words API. In the main thread, assign the randomly generated word to the word property of MainActivity. Possible solution.

   class RandomWordFetcher(
     context: MainActivity,
     private val host: String,
     private val key: String
   ) : AsyncTask<Unit, Unit, String>() {
     private val context = WeakReference(context)
   
     override fun doInBackground(vararg p0: Unit): String {
       val endpoint = "https://wordsapiv1.p.mashape.com/words/"
       val parameters = mapOf("random" to "true", "frequencyMin" to "4", "letters" to "5")
       val url = parameterizeUrl(endpoint, parameters)
       val headers = mapOf("X-RapidAPI-Host" to host, "X-RapidAPI-Key" to key)
       val json = getJson(url, headers)
       val word = json.getString("word")
       return word
     }
   
     override fun onPostExecute(word: String) {
       super.onPostExecute(word)
       context.get()?.let {
         it.word = word
       }
     }
   }
   

4. Define method generateRandomWord in MainActivity to start up a new RandomWordFetcher task. Pass string resources with IDs words_api_host and words_api_key. Possible solution.

   private fun generateRandomWord() {
     RandomWordFetcher(this, resources.getString(R.string.words_api_host), resources.getString(R.string.words_api_key)).execute()
   }
   

5. Define property word in MainActivity to show the word in wordLabel and listen to the user. Possible solution.

   var word: String = ""
     set(value) {
       field = value
       wordLabel.text = value.anagram()
       listen()
     }
   

6. Define method listen to start a new speech recognition activity. Use request code 440. Possible solution.

   private fun listen() {
     // For dialog approach.
     val intent = Intent(RecognizerIntent.ACTION_RECOGNIZE_SPEECH)
     intent.putExtra(RecognizerIntent.EXTRA_PROMPT, wordLabel.text)
     intent.putExtra(RecognizerIntent.EXTRA_LANGUAGE_MODEL, "en-US")
     startActivityForResult(intent, 440)
   
     // For non-dialog approach.
     // recognizer.startListening(recognizeIntent)
   }
   

7. Define method onActivityResult to respond to the results of listening. Grab the candidate utterances and check to see if any of the candidates is a correct unscrambling. Possible solution.

   override fun onActivityResult(requestCode: Int, resultCode: Int, data: Intent?) {
     when (requestCode) {
       440 -> {
         if (resultCode == RESULT_OK && data != null) {
           val candidates = data.getStringArrayListExtra(RecognizerIntent.EXTRA_RESULTS)
           checkCandidates(candidates)
         }
       }
       else -> super.onActivityResult(requestCode, resultCode, data)
     }
   }
   

8. Define method checkCandidates to accept an ArrayList of String. If any element of the list has the same letters as the current word, generate a new word. Otherwise, pop up a toast advising the player to try again and listen to the user for another attempt. Possible solution.

   private fun checkCandidates(candidates: ArrayList<String>) {
     if (candidates.any { word.hasSameLetters(it) }) {
       generateRandomWord()
     } else {
       Toast.makeText(this, "Nope. Try again.", Toast.LENGTH_SHORT).show()
       listen()
     }
   }
   

9. Define field recognitionListener to be an instance of RecognitionListener. If an error occurs, pop up a toast warning. If no error occurs, grab the candidate utterances and check to see if any of the candidates is a correct unscrambling. Possible solution.

   private val recognitionListener = object : RecognitionListener {
     override fun onReadyForSpeech(p0: Bundle?) {}
     override fun onRmsChanged(p0: Float) {}
     override fun onBufferReceived(p0: ByteArray?) {}
     override fun onPartialResults(p0: Bundle?) {}
     override fun onEvent(p0: Int, p1: Bundle?) {}
     override fun onBeginningOfSpeech() {}
     override fun onEndOfSpeech() {}
   
     override fun onError(p0: Int) {
       Toast.makeText(this@MainActivity, "Error. Starting over.", Toast.LENGTH_SHORT).show()
     }
   
     override fun onResults(results: Bundle) {
       results.getStringArrayList(SpeechRecognizer.RESULTS_RECOGNITION)?.let {
         checkCandidates(it)
       }
     }
   }
   

10. Define method initializeSansDialog to create a new SpeechRecognizer that calls back to recognitionListener. Once it’s constructured, generate a new random word. Possible solution.

    private fun initializeSansDialog() {
      recognizeIntent = Intent(RecognizerIntent.ACTION_RECOGNIZE_SPEECH).apply {
        putExtra(RecognizerIntent.EXTRA_LANGUAGE_MODEL, RecognizerIntent.LANGUAGE_MODEL_FREE_FORM)
        putExtra(RecognizerIntent.EXTRA_CALLING_PACKAGE, packageName)
        putExtra(RecognizerIntent.EXTRA_PARTIAL_RESULTS, true)
      }
    
      recognizer = SpeechRecognizer.createSpeechRecognizer(this)
      recognizer.setRecognitionListener(recognitionListener)
    
      generateRandomWord()
    }
    

The full source can be found on GitHub. The master branch contains the final version, and the todo branch has the exercises incomplete.

TODO

Next lecture we will create an app called Two-Face that allows the user to take a split image on the front-facing camera. The left half and right half are taken at separate times. You can achieve some strange effects with such a camera, like a selfie in which you have two tongues, a before and after shot, or a blend of you and your sister.

The UI has 7 widgets:

• a TextureView named previewView that shows the camera preview
• two capture buttons to take a picture and retain just half of it
• two ImageViews that show the captured half-picture
• two reset buttons that dispose of a previously taken half-picture

The exercises don’t require much knowledge of the widgets, but here’s a quick breakdown of the flow. The preview is always active and sits in the background. The two capture buttons are visible initially. As soon as a half-picture is taken, its capture button is hidden, and the ImageView and reset button appear in its place. When reset is hit, the ImageView and reset button are hidden, and the capture button is made visible.

Consult the following resources as you complete your exercises:

• CameraX Overview
• Getting Started With CameraX Codelab
• CameraX Demo App

We’ll start with the following code, which implements the flow described above and finagles the transformations to get the images to appear correctly:

class MainActivity : PermittedActivity() {
  private lateinit var captureUseCase: ImageCapture

  private lateinit var previewView: TextureView
  private lateinit var leftImageView: ImageView
  private lateinit var rightImageView: ImageView

  private lateinit var leftResetButton: ImageButton
  private lateinit var rightResetButton: ImageButton
  private lateinit var leftCaptureButton: ImageButton
  private lateinit var rightCaptureButton: ImageButton

  private var leftBitmap: Bitmap? = null
  private var rightBitmap: Bitmap? = null

  override fun onCreate(savedInstanceState: Bundle?) {
    super.onCreate(savedInstanceState)
    setContentView(R.layout.activity_main)

    leftImageView = findViewById(R.id.leftImageView)
    rightImageView = findViewById(R.id.rightImageView)

    leftResetButton = findViewById(R.id.leftResetButton)
    rightResetButton = findViewById(R.id.rightResetButton)
    leftCaptureButton = findViewById(R.id.leftCaptureButton)
    rightCaptureButton = findViewById(R.id.rightCaptureButton)

    previewView = findViewById(R.id.leftView)

    leftImageView.visibility = View.INVISIBLE
    rightImageView.visibility = View.INVISIBLE
    leftResetButton.visibility = View.INVISIBLE
    rightResetButton.visibility = View.INVISIBLE

    requestPermissions(arrayOf(Manifest.permission.CAMERA, Manifest.permission.WRITE_EXTERNAL_STORAGE), 100, {
      previewView.post {
        initializeUseCases()
      }
      registerCallbacks()
    }, {
      Log.d("FOO", "Bad...")
    })

    window.decorView.systemUiVisibility =
      View.SYSTEM_UI_FLAG_IMMERSIVE_STICKY or
      View.SYSTEM_UI_FLAG_FULLSCREEN or
      View.SYSTEM_UI_FLAG_HIDE_NAVIGATION or
      View.SYSTEM_UI_FLAG_LAYOUT_STABLE or
      View.SYSTEM_UI_FLAG_LAYOUT_HIDE_NAVIGATION or
      View.SYSTEM_UI_FLAG_LAYOUT_FULLSCREEN
  }

  private fun registerCallbacks() {
    leftCaptureButton.setOnClickListener {
      takeLeftImage()
    }

    rightCaptureButton.setOnClickListener {
      takeRightImage()
    }

    leftResetButton.setOnClickListener {
      leftBitmap = null
      syncLeft()
    }

    rightResetButton.setOnClickListener {
      rightBitmap = null
      syncRight()
    }
  }

  private fun sizeToCover(frame: View, image: Bitmap): SizeF {
    val frameAspect = frame.width / frame.height.toFloat()
    val imageAspect = image.width / image.height.toFloat()

    val scaledWidth: Float
    val scaledHeight: Float

    if (frameAspect >= imageAspect) {
      scaledWidth = frame.width.toFloat()
      scaledHeight = scaledWidth / imageAspect
    } else {
      scaledHeight = frame.height.toFloat()
      scaledWidth = scaledHeight * imageAspect
    }

    return SizeF(scaledWidth, scaledHeight)
  }

  private fun coverAnchoredLeft(frame: View, image: Bitmap): Matrix {
    val scaled = sizeToCover(frame, image)
    val xform = Matrix()
    xform.postScale(-1f, 1f, image.width * 0.5f, 0f)
    xform.postScale(scaled.width / image.width.toFloat(), scaled.height / image.height.toFloat())
    return xform
  }

  private fun coverAnchoredRight(frame: View, image: Bitmap): Matrix {
    val scaled = sizeToCover(frame, image)
    val xform = Matrix()
    xform.postScale(-1f, 1f, image.width * 0.5f, 0f)
    xform.postScale(scaled.width / image.width.toFloat(), scaled.height / image.height.toFloat())
    return xform
  }

  private fun updatePreviewTransform(textureSize: Size) {
    val textureAspect = textureSize.height / textureSize.width.toFloat()

    val scaledWidth: Float
    val scaledHeight: Float

    if (previewView.width > previewView.height) {
      scaledHeight = previewView.width.toFloat()
      scaledWidth = previewView.width * textureAspect
    } else {
      scaledHeight = previewView.height.toFloat()
      scaledWidth = previewView.height * textureAspect
    }

    val centerX = previewView.width * 0.5f
    val centerY = previewView.height * 0.5f

    val xform = Matrix()
    xform.postRotate(-viewToRotation(previewView).toFloat(), centerX, centerY)
    xform.preScale(scaledWidth / previewView.width.toFloat(), scaledHeight / previewView.height.toFloat(), centerX, centerY)

    previewView.setTransform(xform)
  }

  private fun syncLeft() {
    if (leftBitmap == null) {
      leftResetButton.visibility = View.INVISIBLE
      leftImageView.visibility = View.INVISIBLE
      leftCaptureButton.visibility = View.VISIBLE
    } else {
      leftCaptureButton.visibility = View.INVISIBLE
      leftImageView.visibility = View.VISIBLE
      leftResetButton.visibility = View.VISIBLE

      leftBitmap?.let {
        leftImageView.setImageBitmap(it)
        leftImageView.imageMatrix = coverAnchoredRight(leftImageView, it)
      }
    }
  }

  private fun syncRight() {
    if (rightBitmap == null) {
      rightResetButton.visibility = View.INVISIBLE
      rightImageView.visibility = View.INVISIBLE
      rightCaptureButton.visibility = View.VISIBLE
    } else {
      rightCaptureButton.visibility = View.INVISIBLE
      rightImageView.visibility = View.VISIBLE
      rightResetButton.visibility = View.VISIBLE

      rightBitmap?.let {
        rightImageView.setImageBitmap(it)
        rightImageView.imageMatrix = coverAnchoredLeft(rightImageView, it)
      }
    }
  }
}

The exercises that you will collectively complete are listed below. Check your email for your assigned exercise and a link to submit your solution. Become an expert on your particular corner of the app, investigating background material as needed. Build on top of the activity and others’ code as you complete your exercise. No solution should be very long, but some have more overhead like curly braces and method signatures than others.

1. Write method createPreviewUseCase that accepts parameters for the screen resolution as a Size and the screen aspect ratio as a Rational. Return a Preview for the front-facing camera, a target resolution that matches the screen resolution, a target aspect ratio that matches the screen aspect ratio, and a rotation that matches the orientation of previewView. Add a OnPreviewOutputUpdateListener to the use case that does three things:
   • Link the SurfaceTexture of previewView to the preview output.
   • Trigger updatePreviewTransform to set up the transformation of the preview. Pass the output’s texture size.
2. Write method initializeUseCases that uses CameraX to bind preview and capture use cases. Compute the screen size and aspect ratio of previewView using the dimensions yielded by preview.display.getRealMetrics.
3. Write method createCaptureUseCase that accepts parameters for the screen resolution as a Size and the screen aspect ratio as a Rational. Return an ImageCapture for the front-facing camera, a target aspect ratio that matches the screen aspect ratio, and a rotation that matches the orientation of previewView.
4. Write method viewToRotation that accepts a View parameter. Examine the view’s display rotation. Return 0, 90, 180, or 270 accordingly.
5. Write method imageProxyToBitmap that accepts an ImageProxy parameter. Return the wrapped image as a Bitmap. (You’ll find more reference code dealing with Image than ImageProxy. Luckily, they have the same interface.)
6. Write method transformBitmap that accepts parameters for a Bitmap and a number of degrees as an Int. Return a new version of the bitmap has been rotated about its center by the specified number of degrees. Use a version of Bitmap.createBitmap that accepts a Matrix.
7. Write method meldBitmaps that accepts parameters for a left Bitmap and a right Bitmap. Return a new bitmap in which the two images appear side-by-side. Use the ARGB_8888 pixel format.
8. Write method saveBitmap that accepts a Bitmap parameter. It stores the Bitmap as a JPEG in the top-level external storage directory under the name twoface.jpg.
9. Write method meldBitmapsAndSaveMaybe that melds the two bitmaps and saves the combined bitmap—but only if both leftBitmap and rightBitmap are valid.
10. Write method takeLeftImage that triggers the captureUseCase to take a picture in-memory. On a successful capture, convert the captured image to a bitmap, close the image, transform it according to the rotation, and crop it to contain only the left half of the image. Update the leftBitmap field, try melding the bitmaps and saving the result, then sync the left half of the UI.
11. Write method takeRightImage that triggers the captureUseCase to take a picture in-memory. On a successful capture, convert the captured image to a bitmap, close the image, transform it according to the rotation, and crop it to contain only the right half of the image. Update the rightBitmap field, try melding the bitmaps and saving the result, then sync the right half of the UI.

See you next time!

Sincerely,

P.S. It’s time for a haiku! I had a hard time finding one that satisfied me, so you get three that didn’t.

It understands me
When I speak, and when I don’t
It undersits me

How to forget things
“Order Mom some flowers, please”
“Make this recurring?”

Share files with SpeechDrop
One phone speaks, the other hears
Now supports images!

Today we will examine two common gestures that occur on mobile platforms: pinch-and-zoom and panning. We won’t create a full app, just a custom view that displays an image and supports these two gestures. First we’ll implement it by just adjusting the bounding rectangle containing the image. Once that works, we’ll switch transform the image with matrices.

Next lecture we will develop an anagram solving app that uses speech recognition to receive the player’s answers. See the TODO below for the assigned exercises.

Consider this extra credit opportunity:

On Friday, 24 May at 1 PM, Chris Johnson will presenting a seminar entitled Computational Making in JE445. Attend and write down a response on a quarter sheet for an extra 0.5 Pakipaki.

ZoomPan

Following are the exercises assigned last time. We will use and discuss the solutions that you’ve submitted as we assemble our app, but I include my solutions here for reference.

1. Define field panListener to be a SimpleOnGestureListener. When a scroll event happens, it shifts the bounding box by the scrolled distance and forces the view to be redrawn. Possible solution.

   private val panListener = object : GestureDetector.SimpleOnGestureListener() {
     override fun onDown(e: MotionEvent): Boolean {
       return true
     }
   
     override fun onScroll(e1: MotionEvent, e2: MotionEvent, distanceX: Float, distanceY: Float): Boolean {
       bounds.offset(-distanceX, -distanceY)
       invalidate()
       return true
     }
   }
   

2. Define field zoomListener to be a SimpleOnScaleGestureListener. When a scale event happens, scale the bounding box. Bonus points if you clamp the scale from becoming too small. Tall poppy points if you scale about fingers’ centroid rather than (0, 0). Possible solution.

   private val zoomListener = object : ScaleGestureDetector.SimpleOnScaleGestureListener() {
     override fun onScale(detector: ScaleGestureDetector): Boolean {
       bounds.left = ((bounds.left - detector.focusX) * detector.scaleFactor) + detector.focusX
       bounds.right = ((bounds.right - detector.focusX) * detector.scaleFactor) + detector.focusX
       bounds.top = ((bounds.top - detector.focusY) * detector.scaleFactor) + detector.focusY
       bounds.bottom = ((bounds.bottom - detector.focusY) * detector.scaleFactor) + detector.focusY
   
       invalidate()
   
       return true
     }
   }
   

3. Investigate gesture detectors in Android. Define fields panDetector and zoomDetector to be detectors for the two listeners defined in the previous steps. Possible solution.

   private val panDetector = GestureDetector(context, panListener)
   private val zoomDetector = ScaleGestureDetector(context, zoomListener)
   

4. Define method fillWidth to set the bounds such that the image fills the view’s width. Set the height to maintain the image’s aspect ratio. Possible solution.

   private fun fillWidth() {
     val imageAspect = image.width / image.height.toFloat()
   
     val fitWidth = width.toFloat()
     val fitHeight = width / imageAspect
   
     val centerX = width * 0.5f
     val centerY = height * 0.5f
   
     bounds.left = centerX - fitWidth * 0.5f
     bounds.right = centerX + fitWidth * 0.5f
     bounds.top = centerY - fitHeight * 0.5f
     bounds.bottom = centerY + fitHeight * 0.5f
   }
   

5. Define method fillHeight to set the bounds such that the image fills the view’s height. Set the width to maintain the image’s aspect ratio. Possible solution.

   private fun fillHeight() {
     val imageAspect = image.width / image.height.toFloat()
   
     val fitHeight = height.toFloat()
     val fitWidth = height * imageAspect
   
     val centerX = width * 0.5f
     val centerY = height * 0.5f
   
     bounds.left = centerX - fitWidth * 0.5f
     bounds.right = centerX + fitWidth * 0.5f
     bounds.top = centerY - fitHeight * 0.5f
     bounds.bottom = centerY + fitHeight * 0.5f
   }
   

6. Define method onSizeChanged to set the bounds to fill the view as much possible—filling either the width or height. Bonus points if you figure out what criteria can fit the image to the view without cropping any of it. Possible solution.

   override fun onSizeChanged(w: Int, h: Int, oldw: Int, oldh: Int) {
     super.onSizeChanged(w, h, oldw, oldh)
   
     val imageAspect = image.width / image.height.toFloat()
     val frameAspect = w / h.toFloat()
   
     if (imageAspect < frameAspect) {
       fillHeight()
     } else {
       fillWidth()
     }
   }
   

7. Define method onDraw to draw the bitmap to fill the current bounds. Possible solution.

   override fun onDraw(canvas: Canvas) {
     canvas.drawBitmap(image, null, bounds, null)
   }
   

8. Define method onTouchEvent to let the detectors process the event. Possible solution.

   override fun onTouchEvent(event: MotionEvent): Boolean {
     zoomDetector.onTouchEvent(event)
     panDetector.onTouchEvent(event)
     return true
   }
   

We are manually tuning the bounding rectangle parameters. Many graphics systems employ matrices to manage these transformations. Let’s see what this app would look like using matrices. First, we need a running transformation, which we declare as an instance variable:

private val xform: Matrix = Matrix()

The panListener gets this onScroll:

override fun onScroll(e1: MotionEvent, e2: MotionEvent, distanceX: Float, distanceY: Float): Boolean {
  xform.postTranslate(-distanceX, -distanceY)
  invalidate()
  return true
}

The zoomListener gets this onScale:

override fun onScale(detector: ScaleGestureDetector): Boolean {
  xform.postTranslate(-detector.focusX, -detector.focusY)
  xform.postScale(detector.scaleFactor, detector.scaleFactor)
  xform.postTranslate(detector.focusX, detector.focusY)

  invalidate()
  return true
}

We want to scale about the centroid of the fingers, so we first subtract off the centroid to make it behave like the origin. Only then do we scale. Because we shifted the centroid, we must also unshift.

Method onSizeChanged must scale and center the image in the viewport. We first shift the image to its center, then we scale it, and then we shift the center to the middle of the viewport.

override fun onSizeChanged(w: Int, h: Int, oldw: Int, oldh: Int) {
  super.onSizeChanged(w, h, oldw, oldh)

  val imageAspect = image.width / image.height.toFloat()
  val frameAspect = w / h.toFloat()

  xform.reset()

  val scale = if (imageAspect < frameAspect) {
    h / image.height.toFloat()
  } else {
    w / image.width.toFloat()
  }

  xform.postTranslate(-0.5f * image.width, -0.5f * image.height)
  xform.postScale(scale, scale)
  xform.postTranslate(0.5f * w, 0.5f * h)
}

Method onDraw uses a different version of drawBitmap:

override fun onDraw(canvas: Canvas) {
  canvas.drawBitmap(image, xform, null)
}

Which version is better? If you are familiar with matrices and their transform semantics, they are generally more versatile—watch our bounding box approach fail when we add rotation. Also, they usually yield more compact code. But for simple needs, matrices certainly aren’t necessary.

TODO

Next lecture we will create an app called Recog that presents anagrams for the user to unscramble. But instead of typing, the user will speak the answer. We’ll use Android’s speech recognition facilities to make this happen. Please read up on speech recognition.

Our main activity will present only a TextView to show the current anagram. We’ll start with this code:

class MainActivity : PermittedActivity() {
  private lateinit var wordLabel: TextView
  private lateinit var recognizer: SpeechRecognizer
  private lateinit var recognizeIntent: Intent

  override fun onCreate(savedInstanceState: Bundle?) {
    super.onCreate(savedInstanceState)
    setContentView(R.layout.activity_main)
    wordLabel = findViewById(R.id.wordLabel)

    generateRandomWord()
  }
}

Random words will be fetched from the Words API web service. They start charging after 2500 requests in a day, so I will not be sharing my key with you. Sorry. Feel free to make your own. Just keep it out of Git.

I have overhauled some code that we wrote earlier in the semester to ease the consumption of their API. Function parameterUrl encodes and appends parameters to a URL for a GET request:

fun parameterizeUrl(url: String, parameters: Map<String, String>): URL {
  val builder = Uri.parse(url).buildUpon()
  parameters.forEach { (key, value) ->
    builder.appendQueryParameter(key, value)
  }
  val uri = builder.build()
  return URL(uri.toString())
}

Function getJson retrieves the JSON from a service request, sending along any extra headers required by the service:

fun getJson(url: URL, headers: Map<String, String>): JSONObject {
  val connection = url.openConnection() as HttpsURLConnection

  headers.forEach { (key, value) ->
    connection.setRequestProperty(key, value)
  }

  try {
    val json = BufferedInputStream(connection.inputStream).readBytes().toString(Charset.defaultCharset())
    return JSONObject(json)
  } finally {
    connection.disconnect()
  }
}

The exercises that you will collectively complete are listed below. Check your email for your assigned exercise and a link to submit your solution. Become an expert on your particular corner of the app, investigating background material as needed. Build on top of the activity and others’ code as you complete your exercise. No solution should be very long, but some have more overhead like curly braces and method signatures than others.

1. Write extension function anagram for String. It generates a new String that is a random shuffling of the letters of the original String. Ensure that the shuffling isn’t identical to the original String.
2. Write extension function hasSameLetters for String that accepts another String as a parameter. It returns true if the two Strings have the same letters. For example, "these".hasSameLetters("sheet") returns true. We need this method because some anagrams may have multiple correct answers, and we want to accept any of them, not just the one we have in mind. For example, reab could unscramble to either bear or bare. Both have the same pronunciation. So it is with break and brake. Can you think of others?
3. Define class RandomWordFetcher to extend AsyncTask and accept a MainActivity, a host string, and a key string as constructor parameters. In the background, fetch the JSON from the Words API endpoint at https://wordsapiv1.p.mashape.com/words/, with the query parameters random=true, frequencyMin=4, and letters=5. Send also the headers X-RapidAPI-Host and X-RapidAPI-Key with the specified strings. The resulting JSON will be formatted according to Words API. In the main thread, assign the randomly generated word to the word property of MainActivity.
4. Define method generateRandomWord in MainActivity to start up a new RandomWordFetcher task. Pass string resources with IDs words_api_host and words_api_key.
5. Define property word in MainActivity to show the word in wordLabel and listen to the user.
6. Define method listen to start a new speech recognition activity. Use request code 440.
7. Define method onActivityResult to respond to the results of listening. Grab the candidate utterances and check to see if any of the candidates is a correct unscrambling.
8. Define method checkCandidates to accept an ArrayList of String. If any element of the list has the same letters as the current word, generate a new word. Otherwise, pop up a toast advising the player to try again and listen to the user for another attempt.
9. Define field recognitionListener to be an instance of RecognitionListener. If an error occurs, pop up a toast warning. If no error occurs, grab the candidate utterances and check to see if any of the candidates is a correct unscrambling.
10. Define method initializeSansDialog to create a new SpeechRecognizer that calls back to recognitionListener. Once it’s constructured, generate a new random word.

See you next time!

Sincerely,

P.S. It’s time for a haiku!

Lazy programmers
Circle extends Rectangle
That’s cutting corners

Today we write an app that solves the problem of choosing of which player goes first in a game. All players place a finger on the phone, and the app chooses one of them randomly. The game is inspired by the commercial app Chwazi.

Next lecture we will discuss writing an app that is multitouch-aware. See the TODO below for the assigned exercises.

Lots

Following are the exercises assigned last time. We will use and discuss the solutions that you’ve submitted as we assemble our app, but I include my solutions here for reference.

1. Declare a PreferenceScreen resource in xml/preferences.xml to present a single EditText preference for the delay in seconds from the last down or up event until one of the active pointers is randomly picked. Make the preference a decimal number with key pickDelay. Give it a title, a summary, a default of 3, and an icon of @drawable/timer. Possible solution.

   <PreferenceScreen xmlns:android="http://schemas.android.com/apk/res/android">
     <EditTextPreference
       android:defaultValue="3"
       android:icon="@drawable/timer"
       android:inputType="numberSigned|number|numberDecimal"
       android:key="pickDelay"
       android:selectAllOnFocus="true"
       android:singleLine="true"
       android:summary="Time in seconds until random player selected"
       android:title="Delay" />
   </PreferenceScreen>
   

2. Define class SettingsFragment as a subclass of PreferenceFragmentCompat that shows the app’s PreferenceScreen. Possible solution.

   class SettingsFragment : PreferenceFragmentCompat() {
     override fun onCreatePreferences(savedInstanceState: Bundle?, rootKey: String?) {
       setPreferencesFromResource(R.xml.preferences, rootKey)
   
       findPreference<EditTextPreference>("delayToPick")?.let {
         it.setOnBindEditTextListener { editor ->
           editor.inputType = InputType.TYPE_CLASS_NUMBER
         }
       }
     }
   }
   

3. Define field pickTask in PickerView to be a Runnable that sets pickedId to a randomly chosen key from pointerIdToPosition. Also, declare field onFirstDown in PickerView to be a lambda that accepts no parameters and returns nothing. Define it to do nothing, but allow clients to redefine it to trigger some action when a first touch event occurs. Possible solution.

   private val pickTask = Runnable {
     pickedId = pointerIdToPosition.keys.random()
   }
   

4. Write method schedulePick that invalidates pickedId and unschedules pickTask from the inherited handler. If there are any entries left in the positions map, schedule pickTask to execute after the pick delay, which you can access via prefs. Watch your units. Possible solution.

   private fun schedulePick() {
     pickedId = -1
     handler.removeCallbacks(pickTask)
     if (pointerIdToPosition.isNotEmpty()) {
       val delay = ((prefs.getString("pickDelay", null) ?: "3").toFloat() * 1000).toLong()
       handler.postDelayed(pickTask, delay)
     }
   }
   

5. Write method handleDowns in PickerView to accept a MotionEvent parameter. Call onFirstDown when the first touch occurs. When any down event occurs, first or otherwise, do two things:
   • Reschedule the picking.
   • Map the active pointer to position (0, 0)—but only if the map doesn’t already have an entry for the pointer.

   Possible solution.

   private fun handleDowns(event: MotionEvent) {
     if (event.actionMasked == MotionEvent.ACTION_DOWN) {
       onFirstDown()
     }
   
     when (event.actionMasked) {
       MotionEvent.ACTION_DOWN,
       MotionEvent.ACTION_POINTER_DOWN -> {
         val id = event.getPointerId(event.actionIndex)
         if (!pointerIdToPosition.containsKey(id)) {
           pointerIdToPosition[id] = PointF()
         }
         schedulePick()
       }
     }
   }
   

6. Write method handlePositions in PickerView to accept a MotionEvent parameter. Iterate through all the currently tracked pointers and update the positions map. Possible solution.

   private fun handlePositions(event: MotionEvent) {
     repeat (event.pointerCount) { i ->
       id = event.getPointerId(i)
       pointerIdToPosition[id]?.apply {
         x = event.getX(i)
         y = event.getY(i)
       }
     }
   }
   

7. Write method handleUps in PickerView to accept a MotionEvent parameter. When any up event occurs, first or otherwise, do two things:
   • Reschedule the picking.
   • Remove any entry for the active pointer from the positions map.

   Possible solution.

   private fun handleUps(event: MotionEvent) {
     when (event.actionMasked) {
       MotionEvent.ACTION_UP, MotionEvent.ACTION_POINTER_UP -> {
         pointerIdToPosition.remove(event.getPointerId(event.actionIndex))
         schedulePick()
       }
     }
   }
   

8. Write method onTouchEvent in PickerView to handle any down events, update the tracked pointer positions, and handle any update events. Also, so that the UI reflects the pointer state, force the view to be redrawn. Possible solution.

   override fun onTouchEvent(event: MotionEvent): Boolean {
     handleDowns(event)
     handlePositions(event)
     handleUps(event)
     invalidate()
     return true
   }
   

9. Write method drawFingerprint in PickerView to accept a Canvas parameter. Draw fingerprint in the center of the view, ensuring that it fits within the bounds. Assume that the fingerprint is square. Possible solution.

   private fun drawFingerprint(canvas: Canvas) {
     val imageAspect = fingerprint.intrinsicWidth / fingerprint.intrinsicHeight.toFloat()
     val frameAspect = width / height.toFloat()
   
     val boxWidth: Int
     val boxHeight: Int
   
     if (frameAspect < imageAspect) {
       boxWidth = width
       boxHeight = (width / imageAspect).toInt()
     } else {
       boxHeight = height
       boxWidth = (height * imageAspect).toInt()
     }
   
     val centerX = (width * 0.5f).toInt()
     val centerY = (height * 0.5f).toInt()
   
     fingerprint.setBounds(centerX - boxWidth / 2, centerY - boxHeight / 2, centerX + boxWidth / 2, centerY + boxHeight / 2)
     fingerprint.draw(canvas)
   }
   

10. Write method onDraw in PickerView to accept a Canvas parameter. If there are no currently tracked pointers, draw the fingerprint as a cue for the user to press the screen. Otherwise, draw circles around each pointer. Use unpickedPaint by default, but if a pointer is the picked pointer, use pickedPaint. Possible solution.

    override fun onDraw(canvas: Canvas) {
      if (pointerIdToPosition.isEmpty()) {
        drawFingerprint(canvas)
      } else {
        pointerIdToPosition.forEach { id, p ->
          canvas.drawCircle(p.x, p.y, 150f, if (id == pickedId) pickedPaint else unpickedPaint)
        }
      }
    }
    

11. Write method enableFullscreen in MainActivity to put the app into fullscreen immersive mode. Ensure that a stable layout is used—even when the navigation controls and status bar are visible. Possible solution.

    private fun enableFullscreen() {
      window.decorView.systemUiVisibility =
        View.SYSTEM_UI_FLAG_IMMERSIVE or
          View.SYSTEM_UI_FLAG_FULLSCREEN or
          View.SYSTEM_UI_FLAG_HIDE_NAVIGATION or
          View.SYSTEM_UI_FLAG_LAYOUT_STABLE or
          View.SYSTEM_UI_FLAG_LAYOUT_HIDE_NAVIGATION or
          View.SYSTEM_UI_FLAG_LAYOUT_FULLSCREEN
    }
    

TODO

Next lecture we will examine two common gestures that occur on mobile platforms: pinch-and-zoom and panning. We won’t create a full app, just a custom view that displays an image and supports these two gestures. Here’s the start to the custom view:

class ZoomPanView @JvmOverloads constructor(context: Context, attributes: AttributeSet? = null, styleAttributes: Int = 0) : View(context, attributes, styleAttributes) {
  private val image: Bitmap = BitmapFactory.decodeResource(resources, R.drawable.some_image)
  private var bounds = RectF()
}

The image will be drawn in the view’s Canvas inside the frame demarcated by bounds. Most of the exercises deal with adjusting bounds appropriately.

The main activity does nothing but load this layout:

<org.twodee.zoompan.ZoomPanView xmlns:android="http://schemas.android.com/apk/res/android"
  xmlns:tools="http://schemas.android.com/tools"
  android:layout_width="match_parent"
  android:layout_height="match_parent"
  android:background="#FFFFFF"
  tools:context=".MainActivity" />

The exercises that you will collectively complete are listed below. Check your email for your assigned exercise and a link to submit your solution. Become an expert on your particular corner of the app, investigating background material as needed. Build on top of the activity and others’ code as you complete your exercise. No solution should be very long, but some have more overhead like curly braces and method signatures than others.

1. Define field panListener to be a SimpleOnGestureListener. When a scroll event happens, it shifts the bounding box by the scrolled distance and forces the view to be redrawn.
2. Define field zoomListener to be a SimpleOnScaleGestureListener. When a scale event happens, scale the bounding box. Bonus points if you clamp the scale from becoming too small. Tall poppy points if you scale about fingers’ centroid rather than (0, 0).
3. Investigate gesture detectors in Android. Define fields panDetector and zoomDetector to be detectors for the two listeners defined in the previous steps.
4. Define method fillWidth to set the bounds such that the image fills the view’s width. Set the height to maintain the image’s aspect ratio.
5. Define method fillHeight to set the bounds such that the image fills the view’s height. Set the width to maintain the image’s aspect ratio.
6. Define method onSizeChanged to set the bounds to fill the view as much possible—filling either the width or height. Bonus points if you figure out what criteria can fit the image to the view without cropping any of it.
7. Define method onDraw to draw the bitmap to fill the current bounds.
8. Define method onTouchEvent to let the detectors process the event.

See you next time!

Sincerely,

P.S. It’s time for a haiku!

It’s not multitouch
When I flit across the phone
It’s finger skating

Today we write an app that allows multiple devices to play a single piano using Google’s Nearby API. One device serves as the host, running a third-party MIDI interpreter. (I use FluidSynth.) A client joins the host. The host device plays octave 4, and the client device plays octave 5.

Next lecture we will discuss writing an app that is multitouch-aware. See the TODO below for the assigned exercises.

P2Piano

Prior to this lecture, I have added the following dependencies to my module’s build.gradle:

implementation 'com.google.android.gms:play-services-nearby:16.0.0'

Following are the exercises assigned last time. We will use and discuss the solutions that you’ve submitted as we assemble our app, but I include my solutions here for reference.

1. Write class Endpoint, which accepts as parameters a string id and a DiscoveredEndpointInfo. Its public interface should expose id, name, and toString. Both name and toString should yield the endpoint’s name—which is not the same as id. Possible solution.

   class Endpoint(val id: String, private val info: DiscoveredEndpointInfo) {
     val name
       get() = info.endpointName
     override fun toString() = name
   }
   

2. In HostActivity, define field payloadListener as a PayloadCallback to receive a MIDI message as an array of bytes. Issue the message with KeyboardActivity.sendMidiMessage. Possible solution.

   private val payloadListener = object : PayloadCallback() {
     override fun onPayloadReceived(id: String, payload: Payload) {
       payload.asBytes()?.let { bytes ->
         sendMidiMessage(bytes)
       }
     }
   
     override fun onPayloadTransferUpdate(id: String, update: PayloadTransferUpdate) {}
   }
   

3. In HostActivity, define field connectionListener as a ConnectionLifecycleCallback. Be prepared to explain the role of its three methods. Automatically accept any attempted connection. The user should know the status of the connection. Communicate events using toasts. Possible solution.

   private val connectionListener = object : ConnectionLifecycleCallback() {
     override fun onConnectionInitiated(id: String, info: ConnectionInfo) {
       Toast.makeText(this@HostActivity, "Connecting to ${info.endpointName}.", Toast.LENGTH_SHORT).show()
       Nearby.getConnectionsClient(this@HostActivity).acceptConnection(id, payloadListener)
     }
   
     override fun onConnectionResult(endpoint: String, result: ConnectionResolution) {
       when (result.status.statusCode) {
         ConnectionsStatusCodes.STATUS_OK -> {
           Toast.makeText(this@HostActivity, "Connected", Toast.LENGTH_SHORT).show()
         }
         ConnectionsStatusCodes.STATUS_CONNECTION_REJECTED -> {
           Toast.makeText(this@HostActivity, "Rejected", Toast.LENGTH_SHORT).show()
         }
         ConnectionsStatusCodes.STATUS_ERROR -> {
           Toast.makeText(this@HostActivity, "Error", Toast.LENGTH_SHORT).show()
         }
       }
     }
   
     override fun onDisconnected(endpoint: String) {
       Toast.makeText(this@HostActivity, "Disconnected", Toast.LENGTH_SHORT).show()
     }
   }
   

4. In HostActivity, define method advertise to start advertising the device as a “server.” The user should know the status of the advertising. Communicate events using toasts. Possible solution.

   private fun advertise() {
     val options = AdvertisingOptions.Builder().setStrategy(Strategy.P2P_POINT_TO_POINT).build()
     Nearby.getConnectionsClient(this).startAdvertising("Chris", "P2Piano", connectionListener, options)
       .addOnSuccessListener {
         Toast.makeText(this, "Advertising...", Toast.LENGTH_LONG).show()
       }.addOnFailureListener {
         Toast.makeText(this, "Failed to advertise...", Toast.LENGTH_LONG).show()
       }
   }
   

5. In JoinActivity, define field payloadListener as a PayloadCallback to do absolutely nothing. The server will not be communicating anything back to the client. Possible solution.

   private val payloadListener = object : PayloadCallback() {
     override fun onPayloadReceived(endpoint: String, payload: Payload) {}
     override fun onPayloadTransferUpdate(endpoint: String, update: PayloadTransferUpdate) {}
   }
   

6. In JoinActivity, define field connectionListener as a ConnectionLifecycleCallback. Be prepared to explain the role of its three methods. Automatically accept any attempted connection. The user should know the status of the connection. Communicate events using toasts. Once a connection has been establishled, retain the endpoint’s ID in the hostEndpoint field. Possible solution.

   private val connectionListener = object : ConnectionLifecycleCallback() {
     override fun onConnectionInitiated(id: String, info: ConnectionInfo) {
       Toast.makeText(this@JoinActivity, "Connecting to ${info.endpointName}.", Toast.LENGTH_SHORT).show()
       Nearby.getConnectionsClient(this@JoinActivity).acceptConnection(id, payloadListener)
     }
   
     override fun onConnectionResult(endpoint: String, result: ConnectionResolution) {
       when (result.status.statusCode) {
         ConnectionsStatusCodes.STATUS_OK -> {
           hostEndpoint = endpoint
           Toast.makeText(this@JoinActivity, "Connected", Toast.LENGTH_SHORT).show()
         }
         ConnectionsStatusCodes.STATUS_CONNECTION_REJECTED -> {
           Toast.makeText(this@JoinActivity, "Rejected", Toast.LENGTH_SHORT).show()
         }
         ConnectionsStatusCodes.STATUS_ERROR -> {
           Toast.makeText(this@JoinActivity, "Error", Toast.LENGTH_SHORT).show()
         }
       }
     }
   
     override fun onDisconnected(endpoint: String) {
       Toast.makeText(this@JoinActivity, "Disconnected", Toast.LENGTH_SHORT).show()
     }
   }
   

7. In JoinActivity, define method joinHost, which accepts an Endpoint. Request a connection to the specified host endpoint. The user should know the status of this request. Communicate events using toasts. Possible solution.

   private fun joinHost(endpoint: Endpoint) {
     Nearby.getConnectionsClient(this@JoinActivity)
       .requestConnection("CLIENT", endpoint.id, connectionListener)
       .addOnSuccessListener {
         Toast.makeText(this@JoinActivity, "Connected to ${endpoint.name}.", Toast.LENGTH_LONG).show()
       }
       .addOnFailureListener {
         Toast.makeText(this@JoinActivity, "Rejected by ${endpoint.name}.", Toast.LENGTH_LONG).show()
       }
   }
   

8. In JoinActivity, define method showEndpointChooser, which accepts a list of Endpoint and returns an ArrayAdapter<Endpoint>. The endpoints in the list are possible hosts that can be joined. The method creates an adapter for the list and then displays it an AlertDialog. When an endpoint from the adapter is clicked on, join it. When the dialog is dismissed, stop discovering new hosts. Possible solution.

   private fun showEndpointChooser(items: List<Endpoint>): ArrayAdapter<Endpoint> {
     val adapter = ArrayAdapter(this, android.R.layout.simple_list_item_1, items)
   
     AlertDialog.Builder(this).run {
       setTitle("Choose host...")
       setAdapter(adapter) { _, i -> joinHost(items[i]) }
       setOnDismissListener {
         Nearby.getConnectionsClient(this@JoinActivity).stopDiscovery()
       }
       show()
     }
   
     return adapter
   }
   

9. In JoinActivity, define method createDiscoverListener, which accepts a MutableList of Endpoint and an ArrayAdapter of Endpoint. It returns an EndpointDiscoveryCallback that adds newly found endpoints to the list and removes lost endpoints from the list, notifying the adapter of any changes. Possible solution.

   private fun createDiscoverListener(items: MutableList<Endpoint>, adapter: ArrayAdapter<Endpoint>): EndpointDiscoveryCallback {
     return object : EndpointDiscoveryCallback() {
       override fun onEndpointFound(id: String, info: DiscoveredEndpointInfo) {
         items.add(Endpoint(id, info))
         adapter.notifyDataSetChanged()
       }
   
       override fun onEndpointLost(p0: String) {
         items.removeIf { it.id == p0 }
         adapter.notifyDataSetChanged()
       }
     }
   }
   

10. In JoinActivity, define method discover. It creates an empty MutableList of Endpoint, pops open an endpoint chooser, and creates a listener for found and lost endpoints. It starts trying to discover hosts. The user should know the status of the discovery. Communicate events using toasts. Possible solution.

    private fun discover() {
      val items = mutableListOf<Endpoint>()
      val adapter = showEndpointChooser(items)
      val callback = createDiscoverListener(items, adapter)
    
      val options = DiscoveryOptions.Builder().setStrategy(Strategy.P2P_POINT_TO_POINT).build()
      Nearby.getConnectionsClient(this).startDiscovery("P2Piano", callback, options).addOnSuccessListener {
        Toast.makeText(this, "Looking for hosts...", Toast.LENGTH_LONG).show()
      }.addOnFailureListener {
        Toast.makeText(this, "Discovery failed!", Toast.LENGTH_LONG).show()
      }
    }
    

11. In JoinActivity, override sendMidiMessage to not only call the superclass version of the method, but also send it to the paired host, if there is one. Possible solution.

    override fun sendMidiMessage(bytes: ByteArray) {
      super.sendMidiMessage(bytes)
      hostEndpoint?.let {
        Nearby.getConnectionsClient(this).sendPayload(it, Payload.fromBytes(bytes))
      }
    }
    

Bluetooth doesn’t work on the emulator, sadly. We will have to test this with two physical devices. My wife lent me her phone for the day. Let’s hope for no emergencies!

TODO

Next lecture we will build an app that solves the problem of choosing of which player goes first in a game. All players place a finger on the phone, and the app chooses one of them randomly. The game is inspired by the commercial app Chwazi.

We will create our own custom View to receive the touch events and draw circles under each finger. To allow a custom View to be included in an XML layout, we must provide certain constructors that will get invoked by the layout inflater. I have added these constructors with some annotation magic in the following PickerView class:

class PickerView @JvmOverloads constructor(context: Context, attributes: AttributeSet? = null, style: Int = 0) : View(context, attributes, style) {
  private var pointerIdToPosition = HashMap<Int, PointF>()
  private var pickedId: Int = -1
  private var fingerprint: Drawable

  private val unpickedPaint: Paint
  private val pickedPaint: Paint
  private val prefs: SharedPreferences

  init {
    unpickedPaint = Paint().apply {
      strokeWidth = 10f
      color = Color.WHITE
      this.style = Paint.Style.STROKE
    }

    pickedPaint = Paint(unpickedPaint).apply {
      color = Color.MAGENTA
      strokeWidth = 20f
    }

    prefs = PreferenceManager.getDefaultSharedPreferences(context)
    fingerprint = resources.getDrawable(R.drawable.fingerprint, null)
  }
}

We will have two activities. The MainActivity shows the PickerView fullscreen and has this layout:

<?xml version="1.0" encoding="utf-8"?>
<org.twodee.lots.PickerView
  xmlns:android="http://schemas.android.com/apk/res/android"
  xmlns:tools="http://schemas.android.com/tools"
  android:id="@+id/picker_view"
  android:layout_width="match_parent"
  android:layout_height="match_parent"
  android:background="@android:color/black"
  tools:context=".MainActivity" />

The SettingsActivity will pop up when the user clicks on a gear icon in the action bar and that has the following layout:

<?xml version="1.0" encoding="utf-8"?>
<fragment
  xmlns:android="http://schemas.android.com/apk/res/android"
  android:id="@+id/settings_fragment"
  android:name="org.twodee.lots.SettingsFragment"
  android:layout_width="match_parent"
  android:layout_height="match_parent" />

The exercises that you will collectively complete are listed below. Check your email for your assigned exercise and a link to submit your solution. Become an expert on your particular corner of the app, investigating background material as needed. Build on top of the activity and others’ code as you complete your exercise. No solution should be very long, but some have more overhead like curly braces and method signatures than others.

1. Declare a PreferenceScreen resource in xml/preferences.xml to present a single EditText preference for the delay in seconds from the last down or up event until one of the active pointers is randomly picked. Make the preference a decimal number with key pickDelay. Give it a title, a summary, a default of 3, and an icon of @drawable/timer.
2. Define class SettingsFragment as a subclass of PreferenceFragmentCompat that shows the app’s PreferenceScreen.
3. Define field pickTask in PickerView to be a Runnable that sets pickedId to a randomly chosen key from pointerIdToPosition. Also, declare field onFirstDown in PickerView to be a lambda that accepts no parameters and returns nothing. Define it to do nothing, but allow clients to redefine it to trigger some action when a first touch event occurs.
4. Write method schedulePick that invalidates pickedId and unschedules pickTask from the inherited handler. If there are any entries left in the positions map, schedule pickTask to execute after the pick delay, which you can access via prefs. Watch your units.
5. Write method handleDowns in PickerView to accept a MotionEvent parameter. Call onFirstDown when the first touch occurs. When any down event occurs, first or otherwise, do two things:
   • Reschedule the picking.
   • Map the active pointer to position (0, 0)—but only if the map doesn’t already have an entry for the pointer.
6. Write method handlePositions in PickerView to accept a MotionEvent parameter. Iterate through all the currently tracked pointers and update the positions map.
7. Write method handleUps in PickerView to accept a MotionEvent parameter. When any up event occurs, first or otherwise, do two things:
   • Reschedule the picking.
   • Remove any entry for the active pointer from the positions map.
8. Write method onTouchEvent in PickerView to handle any down events, update the tracked pointer positions, and handle any update events. Also, so that the UI reflects the pointer state, force the view to be redrawn.
9. Write method drawFingerprint in PickerView to accept a Canvas parameter. Draw fingerprint in the center of the view, ensuring that it fits within the bounds. Assume that the fingerprint is square.
10. Write method onDraw in PickerView to accept a Canvas parameter. If there are no currently tracked pointers, draw the fingerprint as a cue for the user to press the screen. Otherwise, draw circles around each pointer. Use unpickedPaint by default, but if a pointer is the picked pointer, use pickedPaint.
11. Write method enableFullscreen in MainActivity to put the app into fullscreen immersive mode. Ensure that a stable layout is used—even when the navigation controls and status bar are visible.

See you next time!

Sincerely,

P.S. It’s time for a haiku!

Bluetooth’s a weird name
But it got more votes than Flirt
“Close, but not touching”

Today we write an app for hiding and finding messages in geographic locations. Our primary goal is to explore how to acquire and use a device’s location, and this app is just one playful example of the many possible applications that are aware of the user’s place.

Next lecture we will discuss communicating with local devices using Google’s Nearby API. See the TODO below for the assigned exercises.

Hideaway

Prior to this lecture, I have added the following dependencies to my module’s build.gradle:

implementation 'com.google.android.gms:play-services-maps:16.1.0'
implementation 'com.google.maps.android:android-maps-utils:0.5+'
implementation 'com.google.code.gson:gson:2.8.5'

Additionally, I have created a values/google_maps_api.xml file containing an API key granted by Google. Such keys are necessary to access many web services. But they generally should not be included in your VCS repositories. A malicious user could acquire your key and masquerade as your app, possibly racking up fees that will be charged to you as the key owner. This is a vulnerability even for installed apps that store the key. If you plan on publishing an app that relies on an API key, you should do some reading up on the dangers.

Following are the exercises assigned last time. We will use and discuss the solutions that you’ve submitted as we assemble our app, but I include my solutions here for reference.

1. Write class HiddenMessage with a constructor that accepts a string message and a LatLng location. It also creates an empty but mutable list of guessed locations named guesses. All three values are publicly accessible.

   data class HiddenMessage(val message: String, val location: LatLng) {
     val guesses = mutableListOf<LatLng>()
     override fun toString() = "$message | $location | $guesses"
   }
   

2. Write getter-only private property hasLocationPermissions that returns true when the app has been granted the ACCESS_FINE_LOCATION permission.

   private val hasLocationPermissions
     get() = checkSelfPermission(Manifest.permission.ACCESS_FINE_LOCATION) == PackageManager.PERMISSION_GRANTED
   

3. Write method syncUnlockButton to make the unlockMenuItem visible if and only if there’s a hidden message.

   private fun syncUnlock() {
     unlockMenuItem?.isVisible = hiddenMessage != null
   }
   

4. Write method promptForGPS to show a dialog if no GPS provider is enabled. Prompt the user to open the location settings or cancel. Use the simpler Android API, not Google Play Services.

   private fun promptForGPS() {
     val locationManager = getSystemService(Context.LOCATION_SERVICE) as LocationManager
     if (!locationManager.isProviderEnabled(LocationManager.GPS_PROVIDER)) {
       AlertDialog.Builder(this).apply {
         setMessage("GPS is not enabled on your device. Enable it in the location settings.")
         setPositiveButton("Settings") { _, _ ->
           startActivity(Intent(Settings.ACTION_LOCATION_SOURCE_SETTINGS))
         }
         setNegativeButton("Cancel") { _, _ -> }
         show()
       }
     }
   }
   

5. Write method onMapReady, which accepts a GoogleMap parameter that should be accessible later via map. It registers a long-click listener on the map that calls promptForMessage with the pressed location, enables the map’s My Location widgets if the app has location permissions, and calls syncCircles to plot the circles of any hidden message that’s currently being hunted.

   @SuppressLint("MissingPermission")
   override fun onMapReady(googleMap: GoogleMap) {
     map = googleMap
   
     map.setOnMapLongClickListener {
       promptForMessage(it)
     }
   
     if (hasLocationPermissions) {
       map.setMyLocationEnabled(true)
     }
   
     syncCircles()
   }
   

6. Write method hideMessage that accepts a string message and a LatLng location. It tracks the hidden message in hiddenMessage, saves it to a file, synchronizes the unlock button, and synchronizes the circles on the map.

   private fun hideMessage(message: String, location: LatLng) {
     hiddenMessage = HiddenMessage(message, location)
     saveHiddenMessage()
     syncUnlock()
     syncCircles()
   }
   

7. Write method syncCircles that clears the map of any existing circles. If a message is currently hidden, it plots hollow circles for each of the message’s guessed locations. Each circle is centered at its guessed location, and its radius is the number of meters between the guessed location and the location of the hidden message.

   private fun syncCircles() {
     map.clear()
     hiddenMessage?.let {
       it.guesses.forEach { guess ->
         val distance = SphericalUtil.computeDistanceBetween(it.location, guess)
         map.addCircle(CircleOptions().apply {
           center(guess)
           radius(distance)
           strokeColor(Color.RED)
         })
       }
     }
   }
   

8. Write method saveHiddenMessage, which writes out in JSON the current hidden message to hidden.json, a private file on internal storage. Use Gson to produce the JSON. Gson knows how to serialize HiddenMessage without any hints from you.

   private fun saveHiddenMessage() {
     val json = Gson().toJson(hiddenMessage)
     openFileOutput("hidden.json", Context.MODE_PRIVATE).use { writer ->
       writer.write(json.toByteArray())
     }
   }
   

9. Write method promptForMessage that accepts a LatLng location parameter. It prompts the user via an AlertDialog to enter a string. When the user positively dismisses the dialog, it calls hideMessage.

   private fun promptForMessage(location: LatLng) {
     AlertDialog.Builder(this).apply {
       setTitle("What message will you hide here?")
       val view = layoutInflater.inflate(R.layout.message_view, null)
       setView(view)
       setPositiveButton("Hide") { _, _ ->
         val messageEditor: EditText = view.findViewById(R.id.messageEditor)
         hideMessage(messageEditor.text.toString(), location)
       }
       show()
     }
   }
   

10. Write method clearHiddenMessage that removes all traces of any currently hidden message, deleting the file saved earlier by saveHiddenMessage, clearing any circles from the map, and hiding the unlock button from the action bar.

    private fun clearHiddenMessage() {
      hiddenMessage = null
      map.clear()
      deleteFile("hidden.json")
      syncUnlock()
    }
    

11. Write method loadHiddenMessage that loads into hiddenMessage any hidden message previously saved by saveHiddenMessage. Use Gson.

    private fun loadHiddenMessage() {
      try {
        openFileInput("hidden.json").use { reader ->
          val json = String(reader.readBytes())
          hiddenMessage = Gson().fromJson(json, HiddenMessage::class.java)
        }
      } catch (e: FileNotFoundException) {
      }
      syncUnlock()
    }
    

12. Write method unlockMessageSuccess that pops up a dialog showing the hidden message. It prompts the user to either clear the hidden message or leave it in place. Call clearHiddenMessage to clear it.

    private fun unlockMessageSuccess() {
      hiddenMessage?.let {
        AlertDialog.Builder(this).apply {
          setTitle("You unlocked a message!")
          setMessage(it.message)
          setPositiveButton("Clear") { _, _ ->
            clearHiddenMessage()
          }
          setNegativeButton("Leave") { _, _ -> }
          show()
        }
      }
    }
    

13. Write method unlockMessageFailure that accepts a Double distance parameter. It pops up a dialog that tells the user how far away in kilometers they are from the hidden message. Limit the number of digits after the decimal point.

    private fun unlockMessageFailure(distance: Double) {
      AlertDialog.Builder(this).apply {
        setTitle("Too far away")
        setMessage(String.format("You are %.1fk away from the hiddenMessage message.", distance))
        setPositiveButton("OK") { _, _ -> }
        show()
      }
    }
    

14. Write method attemptUnlock that accepts a LatLng parameter for a guessed location. It adds the guess to the current hidden message, synchronizes the circles on the map, and determines the distance to the hidden message. If the distance is under 500m or so, call unlockMessageSuccess. Otherwise, call unlockMessageFailure.

    private fun attemptUnlock(guessLocation: LatLng) {
      hiddenMessage?.let {
        it.guesses.add(guessLocation)
        syncCircles()
        val distance = SphericalUtil.computeDistanceBetween(guessLocation, it.location)
        if (distance < 500.0) {
          unlockMessageSuccess()
        } else {
          unlockMessageFailure(distance)
        }
      }
    }
    

15. Write method queryLocationForUnlock that creates and registers a LocationListener. Listen only for location updates from GPS. As soon a location is received, attempt to unlock the hidden message and unregister the listener.

    @SuppressLint("MissingPermission")
    private fun queryLocationForUnlock() {
      val locationManager = getSystemService(Context.LOCATION_SERVICE) as LocationManager
    
      val listener = object : LocationListener {
        override fun onLocationChanged(location: Location) {
          hiddenMessage?.let {
            attemptUnlock(LatLng(location.latitude, location.longitude))
          }
          locationManager.removeUpdates(this)
        }
    
        override fun onStatusChanged(p0: String?, p1: Int, p2: Bundle?) {}
        override fun onProviderEnabled(p0: String?) {}
        override fun onProviderDisabled(p0: String?) {}
      }
    
      if (hasLocationPermissions) {
        locationManager.requestLocationUpdates(LocationManager.GPS_PROVIDER, 0, 0f, listener)
      }
    }
    

Now we can hide messages in places. The circles on the map perhaps make find the messages too easy. But easy is the point of mobile devices, isn’t it?

TODO

First, please read up on the Nearby Connections API.

Our next app is called P2Piano, which presents a piano interface across two devices and makes use of Nearby. The primary device plays the fourth octave, and the secondary devices plays the fifth octave. I will provide the user interface and the code to generate the music using MIDI commands. I’ll have third-party app FluidSynth running at the same time. It interprets MIDI commands and turns them into sound.

The main activity presents three buttons, one to play the piano solo, one to host a shared piano, and one to join a hosted piano. These buttons will fire up secondary activities. The solo KeyboardActivity knows nothing about other devices. By itself, it only plays music on the local device. You will subclass KeyboardActivity to behave differently when devices are paired. Its implementation is not terribly important, but I include it here for reference:

open class KeyboardActivity : PermittedActivity() {
  private var midiDevice: MidiDevice? = null
  private var port: MidiInputPort? = null
  protected var octave: Int = 4

  override fun onCreate(savedInstanceState: Bundle?) {
    super.onCreate(savedInstanceState)
    setContentView(R.layout.activity_keyboard)

    // The keys are just vanilla Views arranged and colored to look like a
    // keyboard.
    val keys: List<View> = listOf(
      findViewById(R.id.c),
      findViewById(R.id.cd),
      findViewById(R.id.d),
      findViewById(R.id.de),
      findViewById(R.id.e),
      findViewById(R.id.f),
      findViewById(R.id.fg),
      findViewById(R.id.g),
      findViewById(R.id.ga),
      findViewById(R.id.a),
      findViewById(R.id.ab),
      findViewById(R.id.b)
    )

    keys.forEach { key ->
      // Each view has its semitone/halfstep offset set in its tag attribute.
      // C is 0, and B is 11.
      val halfstep = key.tag.toString().toInt()

      // The keys' drawable is a selector that toggles between the key's color
      // when not pressed and cornflower blue when pressed. We must set the
      // selector state explicitly since we are using vanilla views.
      key.setOnTouchListener { view, event ->
        val midiNumber = (12 * (octave + 1) + halfstep).toByte()
        when (event.action) {
          MotionEvent.ACTION_DOWN -> {
            view.isPressed = true
            sendNoteMessage(midiNumber, 127)
            true
          }
          MotionEvent.ACTION_UP -> {
            view.isPressed = false
            sendNoteMessage(midiNumber, 0)
            true
          }
          else -> false
        }
      }
    }
  }

  private fun setupMidi() {
    val midiManager = getSystemService(Context.MIDI_SERVICE) as MidiManager
    val infos = midiManager.devices

    midiManager.openDevice(infos[0], {
      midiDevice = it
      port = it.openInputPort(0)
    }, null)
  }

  override fun onCreateOptionsMenu(menu: Menu): Boolean {
    menuInflater.inflate(R.menu.actionbar, menu)
    return true
  }

  override fun onOptionsItemSelected(item: MenuItem) = when (item.itemId) {
    R.id.setup_midi_button -> {
      setupMidi()
      true
    }
    else -> super.onOptionsItemSelected(item)
  }

  private fun sendNoteMessage(midiNumber: Byte, velocity: Byte) {
    // A note message is three bytes. The most-significant nibble of the first
    // byte is 1001 (0x90) and the least is the channel in 0-15. The second
    // byte is the note's MIDI number. C4 is 60, and C5 is 72. The byte is
    // the velocity or force of the note. 0 means stop playing.
    val buffer = byteArrayOf(0x90.toByte(), midiNumber, velocity)
    sendMidiMessage(buffer)
  }

  protected open fun sendMidiMessage(bytes: ByteArray) {
    port?.send(bytes, 0, bytes.size)
  }
}