CS 491 Lecture 17 – Falling Math
Agenda
- issue camera and gallery intents
- recognize speech
- write a custom speech recognizer
- write a gesture recognizer
Falling Math
Our next app will satisfy several of your requests from your apps wish list: a Space Invaders game, incorporation of novel mobile hardware, and some 2-D drawing stuff (though not much). What’s the app? Well, mathematical expressions are going to fall from the sky. You have to fend them off by solving them. Fun. Learning. But we’re going to solve them with our voices. Or gestures.
- Make a custom view. Override it’s onDraw to paint our falling expression.
- Define it in a layout. Make it full screen.
- Hook it up to a host Activity.
Custom background
A nice feature of mobility is pervasiveness. We can blast expressions anywhere and even integrate our app into the world around us. For instance, we could let the player take a picture of the world being crushed by expressions and use it as our background. Or choose one from the gallery. Let’s pop up a dialog to let them choose where to get a picture. To let them take a new one, we have:
dialog.setPositiveButton("Take one", new DialogInterface.OnClickListener() {
@Override
public void onClick(DialogInterface dialog,
int which) {
Intent photoIntent = new Intent(MediaStore.ACTION_IMAGE_CAPTURE);
photoIntent.putExtra(MediaStore.EXTRA_OUTPUT, photoURI);
startActivityForResult(photoIntent, REQUEST_CAMERA);
}
});
We need that extra there to request the satisfying camera activity to store our image somewhere we can reach it easily—rather than in the Gallery. To choose one from the gallery, we have:
dialog.setNeutralButton("Pick one", new DialogInterface.OnClickListener() {
@Override
public void onClick(DialogInterface dialog,
int which) {
Intent intent = new Intent(Intent.ACTION_PICK, MediaStore.Images.Media.INTERNAL_CONTENT_URI);
intent.setType("image/*");
startActivityForResult(intent, REQUEST_GALLERY);
}
});
ACTION_PICK is a general action that’s used for a whole bunch of things, like contacts, so we must also specify a URI to say just look inside the media gallery. And only look for images.
In the negative case, we just don’t do anything:
dialog.setNegativeButton("No, thanks", new DialogInterface.OnClickListener() {
@Override
public void onClick(DialogInterface dialog,
int which) {
}
});
All right, now we write our hook for when these two activities finish.
private void setBackground(Uri uri) {
try {
Bitmap bitmap = Media.getBitmap(getContentResolver(), uri);
BitmapDrawable drawable = new BitmapDrawable(bitmap);
view.setBackgroundDrawable(drawable);
} catch (IOException e) {
e.printStackTrace();
}
}
public void onActivityResult(int requestCode,
int resultCode,
Intent intent) {
if (requestCode == REQUEST_CAMERA) {
if (resultCode == Activity.RESULT_OK) {
setBackground(photoURI);
cameraFile.delete();
}
} else if (requestCode == REQUEST_GALLERY) {
Log.d("FALL", "picked");
if (resultCode == Activity.RESULT_OK) {
setBackground((Uri) intent.getData());
}
}
}
Speech recognition
Android has had some form of speech recognition since version 1.5, though not all devices may support it. You can ask if it is supported by determining if there’s an intent satisfier installed:
recognizerIntent = new Intent(RecognizerIntent.ACTION_RECOGNIZE_SPEECH);
PackageManager packageManager = getPackageManager();
List<ResolveInfo> activities = packageManager.queryIntentActivities(recognizerIntent, 0);
if (activities.isEmpty()) {
Toast.makeText(this, "Speech recognition not supported.", Toast.LENGTH_SHORT).show();
}
In addition to it being supported, we must also ask the user’s permission to record audio:
<uses-permission android:name="android.permission.RECORD_AUDIO" /> If supported, we can an issue an intent to start recognizing. Like many intents, there are a variety of documented key-value pairs we may need to bundle along. EXTRA_LANGUAGE_MODEL may be LANGUAGE_MODEL_FREE_FORM for open-ended recognition or LANGUAGE_MODEL_WEB_SEARCH for recognition of search queries. The intent satisfier may tweak the recognition algorithm according to this value. EXTRA_PROMPT lets us customize the dialog that pops up.
recognizerIntent.putExtra(RecognizerIntent.EXTRA_LANGUAGE_MODEL, RecognizerIntent.LANGUAGE_MODEL_FREE_FORM);
recognizerIntent.putExtra(RecognizerIntent.EXTRA_PROMPT, "Speak now or forever hold your peace.");
startActivityForResult(recognizerIntent, REQUEST_RECOGNIZE);
The results are available to us in our callback:
if (requestCode == REQUEST_RECOGNIZE) {
if (resultCode == RESULT_OK) {
ArrayList<String> matches = data.getStringArrayListExtra(RecognizerIntent.EXTRA_RESULTS);
Toast.makeText(this, matches.toString(), Toast.LENGTH_LONG).show();
}
}
Okay, this is great and all, but that Activity totally hijacks the interface. That punk. The good news is, the kind folks at Google have (barely) exposed the API so that we can do our own listening in our Activity. It’s only a bit more involved. First, we have another extra we need to provide, which is very poorly documented:
recognizerIntent.putExtra(RecognizerIntent.EXTRA_CALLING_PACKAGE, getClass().getPackage().getName());
Our Activity must also implement the RecognitionListener interface. onResults is where the action is:
@Override
public void onResults(Bundle bundle) {
ArrayList<String> results = bundle.getStringArrayList(SpeechRecognizer.RESULTS_RECOGNITION);
Log.d("FALL", results.toString() + "\n");
}
Of course, there’s no action before we pull the listening trigger. We don’t really want any recognition to happen while the activity is not focused, so the trigger is pulled and released in onResume and onPause:
@Override
public void onResume() {
super.onResume();
recognizer = SpeechRecognizer.createSpeechRecognizer(this);
recognizer.setRecognitionListener(this);
recognizer.startListening(recognizerIntent);
}
@Override
public void onPause() {
super.onPause();
recognizer.stopListening();
recognizer.setRecognitionListener(null);
}
If you want another case of bad documentation, setRecognitionListener’s says null is an unallowable argument. Without it, however, we continue to get listening events.
Okay, now we can get results without losing the screen.
Gestures
One Android version later, gestures came along. With these, your fingers can dance along the screen in custom ways. The first step is to create a library of gestures using the Gesture Builder sample that ships with the Android SDK. Install it on a device or emulator, draw your library, and add the resulting /sdcard/gestures to your raw resource directory. Load it in your Activity with:
gestures = GestureLibraries.fromRawResource(this, R.raw.gestures);
if (!gestures.load()) {
Log.d("FALL", "couldn't load gestures :(");
}
Now, we need to allocate an area of the layout that will receive the gestures. Let’s make the whole screen a gesture overlay:
<android.gesture.GestureOverlayView
xmlns:android="http://schemas.android.com/apk/res/android"
android:id="@+id/gesture_plate"
android:gestureStrokeType="multiple"
android:layout_width="fill_parent"
android:layout_height="fill_parent">
<!-- other views -->
</android.gesture.GestureOverlayView>
The gestureStrokeType attribute must be multiple if gestures involve finger-lifts. And now, we can start listening to gesture events:
GestureOverlayView overlay = (GestureOverlayView) findViewById(R.id.gesture_plate);
overlay.addOnGesturePerformedListener(this);
And handling them:
@Override
public void onGesturePerformed(GestureOverlayView overlay,
Gesture gesture) {
ArrayList<Prediction> predictions = gestures.recognize(gesture);
if (predictions.size() > 0 && predictions.get(0).score > 1.0) {
// predictions.get(0).name is the winner
}
}
Haiku
sign: EMPLOYEES MUST WASH HANDS
BEFORE LOGGING IN