A Curse on Java Bitwise Operators!

Manipulating raw bytes in Java is a major PITA as I was only too painfully reminded a few days ago. I have a few more free days in the next two weeks so I am trying to finish my latest project, an open source editor for an amazing vintage synthesizer, the Casio VZ-1/VZ-10m (a.k.a. Hohner HS-2/HS-2e). Communicating with this synth is done via System Exclusive MIDI messages.

Quick Background Info

It does not matter if you do not have any background on the inner workings of MIDI, for our example the following is all you need:

• MIDI messages start with a byte where the highest bit is set. Such a byte is called a status byte.
• Every byte with that bit set is interpreted to start a new message, therefore the actual data bytes of a message can only use the lowest 7 bits.
• The synthesizer in question (like many others) circumvents this problem when sending data bytes by splitting each data byte into two bytes where the four highest bits are zeroed out:
  0xA7 => 0x0A 0x07

The Original (Buggy) Code

Should be easy, shouldn’t it? Well, this is the method I came up with:

/**
 * builds two nibbles from a byte
 * @param b the byte to divide into nibbles
 * @return two bytes with the lower 4 bits as nibbles of 
 * the input, first hi then lo nibble.
 */
public static byte[] getNibbles(byte b) {
    return new byte[]{
        //move the four high bits to the right, 
        //fill up with zeros 
        (byte)(b >>> 4),
        //zero out the four high bits and leave
        //the low bits untouched
        (byte)(b & 0x0F)
    };        
}

Easy enough, what could possibly go wrong? I use the unsigned right shift operator to move the highest bits to the right and fill up with zeros. The second byte is even easier, I just set the highest bits to zero.

Well, it turns out that my trusted little synth choked on all messages I sent it. Being 25 years old and not exactly developer friendly, all it said was MIDI ERROR. A comparison of the MIDI messages created by my code and by the synth showed that some of the bytes I sent had the high bit set! That was clearly impossible, as all went through the method above, which I have proven to be correct.

Unit Test to the Rescue

Of course when I wrote the method I was 100% sure that this was so easy it could never go wrong, so I was a naughty boy and did not write a test for it.

Well just to verify that this method was not the problem, I wrote a test to prove the problem was somewhere else:

@Test
public void testNibbles() {
    for (int n = Byte.MIN_VALUE; n<=Byte.MAX_VALUE; n++) {
        byte   b        = (byte)n;
        byte[] nibbles  = TransmissionData.getNibbles(b);
        for (byte nibble:nibbles) {
    assertEquals(
                "Byte under test: " + hex(b), 
                hex((byte)(nibble & 0x0F)), hex(nibble));
    }
}

(The hex method just creates a decent hexadecimal representation of a byte)

Wouldn’t you know it, the test failed! The byte created from the highest four bits was starting with four ones for negative bytes:

0xF4 => 0xFF 0x04

So the bug was in the following line:

//move the four high bits to the right, 
//fill up with zeros 
(byte)(b >>> 4),    

To understand what went wrong we need to remember that all Java bitwise operators are performed on ints. So this is what happens in the VM:

• turn b into an int
• shift the resulting int 4 bits to the right & fill up with zeros
• turn the result into a byte by truncating the first 24 bits

Still, this sounds OK. The problem however is that the promotion of a byte into an int preserves the sign! So this is what happens to 0xF4:

0xF4 => 
/*promote to int*/
0xFFFFFFF4 => 
/*unsigned shift*/
0x0FFFFFFF => 
/*downcast to byte*/
0xFF

The solution was to simply bitwise and all higher bits on the int before shifting:

(byte)((b & 0xFF) >>> 4)

Now the following happens:

0xF4 => 
/*promote to int*/
0xFFFFFFF4 => 
/*bitwise and*/
0x000000F4 =>
/*unsigned shift*/
0x0000000F => 
/*downcast to byte*/
0x0F

Conclusions

• If you want to bitwise modify bytes, always, always bitwise and them first: b & 0xFF!
• The combination of only signed bytes and the signed upcast of byte to int for bitwise operators makes byte manipulation in Java a painful experience.

Update: Most important conclusion: Make sure the critical functionality of your app is covered by unit tests right from the start!