The students were using BufferedImage and ImageIO in a homework assignment that I should have called SimFur, as they were generating animal skin patterns using cellular automata. One student hardcoded a path that would only have worked on the student’s computer. Consider this minimum working example that demonstrates the issue:

import java.awt.image.BufferedImage;
import java.io.IOException;
import java.io.File;
import javax.imageio.ImageIO;

public class Demonstration {
  public static void main(String[] args) throws IOException {
    BufferedImage image = new BufferedImage(300, 200, BufferedImage.TYPE_INT_RGB);
    File file = new File("/nonexistent/directory/bad/bad/bad.png");
    ImageIO.write(image, "png", file); 
  }
}

When I graded the student’s work using my unit tests, this was the error message that popped up:

Exception in thread "main" java.lang.NullPointerException
        at javax.imageio.ImageIO.write(ImageIO.java:1538)
        at Demonstration.main(Demonstration.java:10)

This is not as helpful a message as it could be, and the student didn’t understand what the issue was. What makes matter worse is that when I ran this code outside my unit tests, I got this output:

java.io.FileNotFoundException: /nonexisting/directory/bad/bad/bad.png (No such file or directory)
        at java.io.RandomAccessFile.open0(Native Method)
        at java.io.RandomAccessFile.open(RandomAccessFile.java:316)
        at java.io.RandomAccessFile.<init>(RandomAccessFile.java:243)
        at javax.imageio.stream.FileImageOutputStream.<init>(FileImageOutputStream.java:69)
        at com.sun.imageio.spi.FileImageOutputStreamSpi.createOutputStreamInstance(FileImageOutputStreamSpi.java:55)
        at javax.imageio.ImageIO.createImageOutputStream(ImageIO.java:419)
        at javax.imageio.ImageIO.write(ImageIO.java:1530)
        at Demonstration.main(Demonstration.java:10)
Exception in thread "main" java.lang.NullPointerException
        at javax.imageio.ImageIO.write(ImageIO.java:1538)
        at Demonstration.main(Demonstration.java:10)

See that helpful error message, with the FileNotFoundException? That’s the one I wanted my student to see. But for some reason or another I got the NullPointerException instead. I sensed that somewhere the FileNotFoundException was getting caught and printed to System.err instead of bubbling up to my handling code. In its place came the less meaningful NullPointerException.

Since the NullPointerException arose in ImageIO.write, I figured the standard library must be doing this suppression. I went to the OpenJDK source code and worked backwards from ImageIO.write. The offending code was in FileImageOutputStreamSpi:

try {
  return new FileImageOutputStream((File)output);
} catch (Exception e) {
  e.printStackTrace();
  return null;
}

The standard library should not be doing output like this! Why even have exceptions if this is how they are going to be handled?! I can accept the excuse that maybe the method wasn’t suppose to throw any checked exceptions, but then it should have been implemented this way:

try {
  return new FileImageOutputStream((File)output);
} catch (Exception e) {
  throw new RuntimeException(e);
}

Wrapping the checked exception up in an unchecked exception would at least have allowed meaningful information about the error to propagate up the call chain.

My disappointment was somewhat abated when I saw that there was an open issue to get this fixed. Then I saw that it was first reported in April 2004. Twelve years later, no progress has been made. According to the comments, this method was never even specified to return null. To add salt to the wound, I was unable to vote up the issue because Joe Users like me can’t get accounts on their issue tracker.

tokens = lex src
for each token in tokens
  case token.type
    when COMMENT
      color src[token.start..token.end] in mute gray
    when NUMBER
      color src[token.start..token.end] token orange
    when IDENTIFIER
      color src[token.start..token.end] token dark blue
    etc
  end
end

It worked pretty well, except the GUI framework complained about the indices of the last couple of tokens. Oh, yeah. The last couple of tokens are artificial ones that aren’t really in the original source. The last one is END_OF_FILE, which I emit so the parser knows when to stop processing. The second to last one is NEWLINE, which I insert just in the case the programmer didn’t include one. If the parser can assume that all statements end with a NEWLINE, the rules are simpler. (This practice is common. See how POSIX defines line.) Trying to colorize these artificial tokens triggered the warning.

So, I changed my loop a bit:

for i = 0; i < tokens.size - 2; ++i
  ...
end

This generated a segfault—sometimes. My routine worked fine on blank source. It worked on longer programs too. But sometimes it crashed. That’s weird, I thought. I’m doing strictly less work than before. How can I be getting a segfault? I wish I could say I found the answer right away. But I didn’t. Here’s a minimum working example that demonstrates the same problem:

#include <iostream>
#include <vector>

int main(int argc, char **argv) {
  std::vector<int> numbers;
  for (int i = 0; i < numbers.size() - 2; ++i) {
    std::cout << "numbers[" << i << "]: " << numbers[i] << std::endl;
  }
  return 0;
}

On my machine, this generates a segfault pretty reliably. It might not on your machine, but it definitely shouldn’t print anything, since the vector is empty.

The issue is numbers.size() - 2. The size method probably returns a size_t value, an unsigned type. If the size is ever 0, the subtraction will wrap around and the upper bound will be really big:

#include <iostream>

int main(int argc, char **argv) {
  size_t size = 0;

  // prints 18446744073709551614
  std::cout << "size - 2: " << size - 2 << std::endl;

  return 0;
}

So, how was the number of tokens ever 0, given my two artificial tokens? When the lexer encounters a unknown token early in the processing, I raise an exception and no tokens are emitted. I still want the highlighter to colorize as much as it can on such an exception, so I must still process whatever tokens I do have.

The fix was simple: just favor addition over subtraction:

#include <iostream>
#include <vector>

int main(int argc, char **argv) {
  std::vector<int> numbers;
  for (int i = 0; i + 2 < numbers.size(); ++i) {
    std::cout << "numbers[" << i << "]: " << numbers[i] << std::endl;
  }
  return 0;
}

Because I suffer from developerism, I asked my wife if she’d let me try and generate some calendar pages for her. She agreed and even sat and watched me code up the script that I share below. She is rarely the direct benefactor of my career and skill set, and we will forever treasure this night when she watched me code up a calendar generator. Especially when we run it each year for the rest of our lives.

A page from the end results looks some like this:

The generator is written in Ruby and its output is LaTeX source, which can be fed into a LaTeX interpreter like pdflatex.

/usr/lib/ruby/vendor_ruby/rubygems/dependency.rb:315:in `to_specs': Could not find 'coderay' (>= 0) among 94 total gem(s) (Gem::MissingSpecError)
Checked in 'GEM_PATH=/var/www/.local/share/gem/ruby/3.2.0:/var/lib/gems/3.2.0:/usr/local/lib/ruby/gems/3.2.0:/usr/lib/ruby/gems/3.2.0:/usr/lib/x86_64-linux-gnu/ruby/gems/3.2.0:/usr/share/rubygems-integration/3.2.0:/usr/share/rubygems-integration/all:/usr/lib/x86_64-linux-gnu/rubygems-integration/3.2.0:/home/johnch/.gems' , execute `gem env` for more information
	from /usr/lib/ruby/vendor_ruby/rubygems/dependency.rb:327:in `to_spec'
	from /usr/lib/ruby/vendor_ruby/rubygems/core_ext/kernel_gem.rb:57:in `gem'
	from ./coderay:24:in `
'

Happy new year!

ssh tester@localhost "cd student/folder && forn 30 theirs.exe"

The script works fine when not run across SSH. I don’t know what the problem is. Instead of asking the Internet for help, I just rewrote the script in Ruby, and it works just fine across SSH:

#!/usr/bin/env ruby

# ---------------------------------------------------------------------------- 
# FILE:   forn                                                                 
# AUTHOR: Chris Johnson                                                        
# DATE:   Dec 03 2014                                                          
#                                                                              
# Executes a command with a time limit. If the command takes longer than the
# specified number of seconds, it is killed and status 37 is returned.
# Otherwise, the exit status of the command is returned.
# ---------------------------------------------------------------------------- 

require 'timeout'

if ARGV.length < 1
  puts "Usage: forn delay-in-seconds command arg1 arg2 ..."
  exit 1
end

delay = ARGV[0].to_i
ARGV.shift

# Start command, but kill it if delay elapses.
pid = Process.spawn(ARGV.shelljoin)
begin
  Timeout.timeout(delay) do
    Process.wait pid
    exit $?.exitstatus
  end
rescue Timeout::Error
  puts "Uh oh. Execution timed out after #{delay} second#{delay == 1 ? '' : 's'}."
  Process.kill(:SIGTERM, pid)
  exit 37
end

Yesterday I was browsing Allan Martin’s Teaching and Learning with Logo. In chapter 9, Making Music with Logo, I stumbled across this bug report:

A problem which may arise, especially when long tunes with harmony are being played, is that inexplicable pauses may suddenly interrupt the music.

Inexplicable pauses? What could cause those? I read on.

These are due to “garbage collections”: these take place whenever the computer is running short of memory to run a program, and involve a reorganisation of items in the computer’s memory in order to make as much memory as possible available to run the program. Automatic garbage collections are part of all LISP and LOGO systems. They are essential, since these systems tend to use up a lot of memory whilst running programs; and garbage collections maximise the amount of memory available. But they do produce a short pause in the program run, and can occur at inconvenient moments.

The bug was actually a feature: automatic garbage collection. Instead of asking the developer to explicitly free data that is no longer needed, the runtime periodically checks for data that can no longer be accessed and frees it. However, it’s hard to completely hide low-level implementation details under an abstraction. When these details bubble up, knowledge and control of the hidden behavior becomes essential, as Martin documents:

To avoid trouble in musical pieces, the easiest path is to include a command which will cause a garbage collection to take place in the program. This should be done at the beginning and, if it’s a long piece, at further convenient points, such as the beginnings and ends of phrases, where a very short pause will seem natural.

Instead of asking the IDE to reveal context, we can have the language itself do so. Objective-C, Smalltalk, Python, Ada, Scala, SQL, R, and others do this through named parameters. In Python, we can define the ellipse function like so:

def ellipse(x, y, width, height):
  ...

We may then call the function with an undecorated sequence of positional parameter values, or we may name each one, possibly reordering them:

ellipse(10, 100, 40, 60)
ellipse(x=10, y=100, height=60, width=40)

While I prefer having the language reveal the context instead of the IDE, named parameters do not always contribute value. Sometimes what I want to pass as a parameter is already stored in a meaningfully named variable. In Objective-C, where the names are not optional, passing a message to an object with like-named variables feels silly:

[[Database sharedDatabase]
  addStudentWithFirstName:firstName 
                 lastName:lastName 
                 username:username 
                 inCourse:self.course];

I’m thinking of adding mandatory named parameters to Madeup, but I’d like to avoid the sillyness of labeling z when it’s already named z. One idea that I’ve been considering is to allow developers to omit the explicit mention of z if there’s already a variable in the environment named z. That is, instead of expressing

moveto x:10 y:30 z:0
moveto x:20 y:40 z:0

we allow for

z = 0
moveto x:10 y:30
moveto x:20 y:40

At the time of a function call, if a named parameter is not explicitly passed, we look up the name in the current bindings to determine the value to pass. I’m not sure. One could argue that I’d be hiding the interface of a function. Default named parameters already do this. Plus, the IDE could optionally show the implicit parameter. The benefit is that a developer doesn’t have to express her intent multiple times. “This is what I want z to be.” And that’s that.

Good idea or bad idea?

I see after writing this that Scala already supports this idea.

When we arrived, I learned that there was zero infrastructure for judging the teams’ submissions. That meant I had to roll my sleeves up and starting writing a shell script to pull a team’s code off a USB drive, compile it, run it with the test cases, and diff the actual output with the expected output. I wrote the script, it worked great, and I felt like a true hero.

However, sometimes the teams’ code would take longer than the allotted time to compute an answer. I really wanted my script to run their code for no more than N seconds. It didn’t happen that day. Several months later, I now have a little utility that does just that.

#!/usr/bin/env zsh

# ---------------------------------------------------------------------------- 
# FILE:   forn                                                              
# AUTHOR: Chris Johnson                                                        
# DATE:   Dec 29 2013                                                          
#                                                                              
# Runs a shell command for at most a specified number of seconds. If the job
# takes longer than the allotted time, it's killed and the script exits with
# status 143. Otherwise, the job finishes normally or fails before the timer
# expires. In either case, this script exits with the job's exit status.
#
# The long-running job does not accept input interactively.
#
# Usage:
#   forn nseconds command [arg1 [arg2 ...]]
#
# Examples:
#   forn 3 sleep 5
#   forn 90 java MyClass -g -a -l case006.txt
#   forn 10 sort < names.txt
# ---------------------------------------------------------------------------- 

if [[ $# -lt 2 ]]; then
  echo "Usage: $0 nseconds command [arg1 [arg2 ...]]" >&2
  exit 1
fi

nseconds=$1
shift

# cmd is going to be embedded in a string, so we'll need to do some quoting of
# its elements for correct interpolation.
cmd=(${(q-)@})

# Kills the specified process after nseconds have expired.
sleepkill() {
  sleep $1
  kill $2
}

# Start up both the long-running process and a sleep timer. The long-running
# process is made up of ARGV[2..]
eval "($cmd) &"
longpid=$!

sleepkill $nseconds $longpid &
sleeppid=$!

# By default, Control-C will kill the wait that happens below -- and not
# longpid. I want it to kill longpid and the sleep timer.
TRAPINT() {
  kill $longpid
  kill -HUP -$$
}

# Wait for longpid to finish. If longpid has already finished, wait will flash
# a message saying it doesn't know about the process. I don't want to see that
# message.
wait $longpid 2>/dev/null

# If longpid's exit status is > 128, then a signal was sent to it to kill it.
# If that signal was SIGTERM/15, then we'll see a status of 143 (128 + 15).
ok=$?
if [[ $ok -eq 143 ]]; then
  print "Command $cmd timed out."
fi

# If longpid finishes before the sleep timer, let's kill the sleep timer.
kill $sleeppid 2>/dev/null

exit $ok

2015, here we come.

1. He creates one source repository on Bitbucket for an assignment, committing to it any scaffolding code that he provides to students.
2. Each student forks his repository. The student makes the fork private, so only he and the student can see it.
3. The students work work work.
4. My friend clones each student’s repository at the homework deadline. Whatever was last pushed gets graded.

I think this idea is great. I’d been using a submission directory on our students’ file server. This method had its deficiencies: students worked in one directory but submitted to another, causing synchronization confusion; students’ code was locked up in university storage behind a VPN; and reliably offering read and write access to the share for both Windows and Linux was as likely to happen as us getting a new building.

This next semester I will be adopting my friend’s approach. The only thing I saw missing was a way to get a list of all the students’ repositories to make checkouts quick and painless. The following Ruby script was my solution. Given a Bitbucket user and the name of repository owned by that user, it collects the names of all the repository’s forks using Bitbucket’s API and assembles them in a simple CSV format. The script uses basic HTTP authentication to gain access to the user’s repository metadata.

#!/usr/bin/env ruby

# ---------------------------------------------------------------------------- 
# FILE:   bbforks                                                              
# AUTHOR: Chris Johnson                                                        
# DATE:   Dec 17 2013                                                          
#                                                                              
# A script for amassing the names of all forks of a Bitbucket repository and
# exporting them in CSV format.
# ---------------------------------------------------------------------------- 

require 'net/http'
require 'net/https'
require 'json'

if ARGV.length != 2
  STDERR.puts "Usage: #{$0} user repo"
  exit 1
end

user = ARGV[0]
repo_slug = ARGV[1]
nresults_per_page = 100
page_link = "https://bitbucket.org/api/2.0/repositories/#{user}/#{repo_slug}/forks?pagelen=#{nresults_per_page}"

# Get the password of owner of original repository. Do not echo password
# characters to the console.
`stty -echo`
STDERR.print 'Password: '
password = $stdin.gets.chomp
`stty echo`
STDERR.puts ''

forks = []

# Grab the pages of fork listings from Bitbucket. A single request may not grab
# all forks, as Bitbucket paginates the result. A single page has the following
# form:
#
#   {
#     values: [
#       {
#         full_name: 'forking-user/forked-repo-name',
#         is_private: <true-or-false>,
#         owner: {
#           display_name: 'Firstname Lastname',
#           ...
#         },
#         ...
#       },
#       <fork2>,
#       <fork3>,
#       ...
#     ],
#     next: <link-to-next-page-of-forks>,
#     ...
#   }
#
# If there is no next page, key next will not appear.
#
# Currently, we retain only the path to each repo and the owners' names.
while page_link
  uri = URI page_link
  Net::HTTP.start(uri.host, uri.port, :use_ssl => true, :verify_mode => OpenSSL::SSL::VERIFY_NONE) do |http|
    request = Net::HTTP::Get.new uri.request_uri
    request.basic_auth(user, password)
    response = http.request request

    body = JSON.parse response.body
    page_link = body[:next]

    body['values'].each do |fork|
      STDERR.puts "#{fork['full_name']} is not private!" if !fork['is_private']
      forks << {repo: fork['full_name'], owner: fork['owner']['display_name']}
    end
  end
end

# Emit the listing of forks in CSV.
puts "owner,repository"
forks.sort_by {|fork| :repo}.each {|fork| puts "#{fork[:owner]},#{fork[:repo]}"}

1. I want to be able to run Mac bundles from the Terminal, such that standard in and out stay in Terminal and that the application window is frontmost.

OS X provides the utility open, but open runs the bundle asynchronously, and we lose standard in and standard out from Terminal. Historically, standard out used to go to Console.app, but that’s not true anymore. Alternatively, we can run the executable inside the bundle directly to retain standard in and out, but then the application window is not brought to the foreground.

Instead of waiting for a myth to come and fix the operating system for me, I took matters into my own hands. Here’s a Ruby script that replaces the open utility, but keeps standard in and standard out in the Terminal and puts the application window in front:

#!/usr/bin/env ruby

# ---------------------------------------------------------------------------- 
# FILE:   runx                                                                 
# AUTHOR: Chris Johnson                                                        
# DATE:   Dec 16 2013                                                          
#                                                                              
# A script for starting a GUI app in Mac OS X from Terminal. Compared to the
# traditional "open myapp.app", this script runs the app synchronously, keeps
# its output and input in the terminal, and foregrounds the application
# window.
#
# Usage:
#   runx path/to/app/bundle.app [args]
# ---------------------------------------------------------------------------- 

require 'pathname'

DELAY_BETWEEN_CHECKS = 1.0

if ARGV.length == 0 || ARGV[0] !~ /\.app$/
  puts "Usage: #{$0} path/to/app/bundle.app [args]"
  exit 1
end

app = ARGV.shift
exe = Pathname.new(app).basename.sub_ext('').to_s

foreground_script = <<EOF
-- Activate the application to bring it into the foreground. Returns "yes" if
-- successful and "no" otherwise.

-- A blind activate call causes the application to open if it's not opened
-- yet. We only want to communicate with an instance of the application that's
-- already open.
if application "#{app}" is running then
  try
    tell application "#{app}" to activate
    return "yes"

  -- Sadly, even if the application is open, it may not be ready to respond
  -- to activate. I was seeing these errors occasionally: "execution error:
  -- #{app} got an error: Application isn't running. (-600)" even if the
  -- condition above was met.
  on error message number errno
    return "no"
  end try
else
  return "no"
end if
EOF

# Make a side process that waits for the application to be foregroundable. Once
# it is, let's foreground it and be done.
fork do
  is_foregrounded = false

  # If the application's process died for some reason, we became a zombie. Our
  # parent process is init, which has pid 1. There's no point in continuing if
  # the parent is dead.
  while Process.ppid != 1 && !is_foregrounded
    sleep DELAY_BETWEEN_CHECKS
    IO.popen('osascript', 'r+') do |io|
      io.write(foreground_script)
      io.close_write
      is_foregrounded = io.gets.chomp == 'yes'
    end
  end
end

# Replace this process with the application. We reach inside the bundle and
# invoke the executable directly so that stdout and stdin come from the
# terminal. If we ran "open $app" instead, the process would detach from
# the terminal.
exec "#{app}/Contents/MacOS/#{exe} #{ARGV.join(' ')}"

./autograde "Homework 2" /path/to/submissions

Or maybe I just want to grade Quintin Buster’s submission. I just supply his username as a parameter:

./autograde "Homework 2" /path/to/submissions BUSTERQ

I’ve had an itch for a very long time to enable autocompletion of my students’ usernames, which I can almost never remember when I’m grading. I’ve now scratched that itch with zsh using compctl:

_ids() {
  reply=(BUSTERQ USERNAME2 USERNAME3 USERNAME4)
}

compctl -f -K _ids autograde

This compctl command adds completion for the autograde script. The -f says to complete file names as parameters to autograde. The -K _ids says to also add the results of calling the _ids functions to the list of things that can be completed.

I threw these commands into a script in my gradebook directory, which I source when I’m ready to grade. Instead of assigning to reply a list of literal usernames, I read the usernames from a file I store for each class I teach.