Beat Box Project Evaluation (PART 1)

Jargon


TUIO - Stands for Tangible User Interface Object. 


MIDI - Stands for Musical Instrument Digital Interface


Fiducial - In this case, a fiducial is a symbol that is recognized by and interacts with the ReacTIVision software.



1.Problem Solving and methods of working:

Whilst working on the beat box project I found that I needed to upgrade my working methods and knowledge by introducing a lot of new research and interactive techniques. This was crucial for not only my project planning but also for my idea and skill development. I had to start from scratch with every piece of software and hardware that I have used throughout the past 6 months, ranging from computer programs to infrared lighting and diffuse illumination. Problems occurred throughout the project and were thrown at me from every angle. There were only 2 problems that I came across that really caused me any bother with the construction and creation of the Beat Box.

The first major issue I came across was the link I needed to create to connect the relevant programs. It took me a while to decide on which audio software I was going to be comfortable with in order to produce the adequate sounds for the table application. It was a case of trial and error and seeing what I could achieve within the programs. I started by experimenting in Ableton Live by using samples and trying to sort them into suitable functional sections.

These sections were to be assigned to each fiducial shape that is placed on the table surface. In order to link ReacTIVision with Ableton live I was to use the 3dmix max patch for Max Msp, which allows you to assign movement functions to each fiducial, such as vertical and horizontal faders, Control knobs and on/off states. The problem that I encountered was the link itself between ReacTIVision, Max Msp and Ableton which involves ReacTIVision sending TUIO (Table Top User Interfaces) which has to be converted in Midi signals in order for Ableton Live to receive and process the signals. It was this stage that my problems occurred and I just couldn’t seem to link the programs.

I then decided to try an alternative approach and concentrate on using Reason 4 instead of Ableton Live. It took another long period of time to even get to grips with the way in which reason works. Its one of those programs that looks relatively simple but really isn’t. The creators of reason have decided to make it as realistic to an actually amp rack as possible. This means the user has to create yet more links by flipping the rack around and physically wiring up the individual racks to effects etc. This made my job a lot harder but I managed to pull through.

Mapping fiducial functions In Midi rather than TUIO proved to be a lot easier in the program but required a lot of coding. To do this you have to edit an external XML file called MIDI.xml. This file is a command file and where in MAX MSP you can simply map using a patch you have to input commands such as:





Looking at the image you can clearly see which piece of code is relevant to each function.

After much playing about with reason I managed to assign fiducials to on/off functions on individual samples along with rotational knobs to control effects. You can find this in a preview Blog post (Sunday 24th January).

Midi seemed so promising but I had then realised “how am I to link the surface visuals using MIDI?”

It just wasn’t possible so I was to yet again re think my strategy if I was to create the perfect program link up to power the Beat Box. I reverted back to Ableton to chase TUIO progress. In the end I was able to create the link by accessing and setting up the IAC bus driver within the iMac preferences. An IAC Bus is a virtual MIDI bus that allows MIDI communication between applications. This enabled ReacTIVision and max Msp to link up via port 3333, which is the default port in which ReacTIVision processes its data. After sorting these links out I was then able to go on the assign all 14 cubes containing 86 samples altogether and have them working on the surface of the table.

Below is an image of Ableton live during the sample assigning process:




Below is an image of the table in its earliest testing form once the samples had been assigned:





The second difficulty that I ran into whilst creating the Beat Box was involving the Playstation 3 Pseye camera. I originally wanted to use two different spectrums of light inside the box in order for the projections to be seen clearly on the surface. I had the idea all planned out and I began to experiment by taking apart the pseye camera and extracting its infrared filter lens and replacing it with a piece of floppy disk. The floppy disk acted as a new filter, which cut out the light from the projector. This helps to give an even spread of light onto the underside of the table surface when using infrared. I then found out this method was yet to be used with fiducials and cubes due to not having an infrared reflective surface on the cubes. This caused a huge error and I had to re visit the normal camera solution with standard light, but to do this I had to spend out on a second camera in order for it to work. This put a hole in the bank funds but in the long run I now have an infrared camera for if I wish to look into multitouch finger tracking applications.


2.How the product works:

The Beat Box is a tangible sound device, which relies heavily on sound channels and processing. Users use blocks on the surface in order to change and interact with sound samples. Once a cube is placed down the shape of the sound can be seen on the surface projections, which take the form of a moving graphic equalizer. To be able to use the beat box a specific set up is needed in order for everything to work. This can either be acquired from myself or you can follow the small tutorial for the initial set up that I have included in this section.

Below is a list of the programs I eventually used with their corresponding functions and uses.

Software Purposes


ReacTIVision: ReacTIVision is a computer vision framework for the fast and robust tracking of fiducial markers via a camera. 


TUIO Simulator: A java application that simulates ReacTIVision, i.e. it simulates a real table so that you can configure all of the software without the need of a camera and a physical table set up.


MIDI Monitor: An application that intercepts all data being broadcast over the MIDI controller channel and notes what information is being sent. This is useful to monitor exactly what is going on between Max/MSP and Ableton Live. 


3dMix patch: A kind of application that runs inside Max/MSP because Max/MSP alone is useless, it needs to be instructed, that is what this patch does. 


Ableton Live: Ableton Live is versatile loop-based music sequencer software for Mac or PC. Used to produce and manage the sound samples.


Max/MSP: Max/MSP is a graphical development environment for music and multimedia. 


SoundflowerBed: SoundflowerBed is a Mac OS X (10.2 and later) system extension that allows applications to pass audio to other applications. It was used in The Beat Box project to create a link between Ableton live and Quartz Composer in order for the graphic equalizer to pick up and move to the incoming Midi signals.

Quartz Composer: Quartz Composer is a groundbreaking graphics development environment that allows you to explore the incredible power of the graphics stack of Mac OS X. Used to create the visuals for the table surface projection.


Once all the above software is acquired the necessary audio setup needs to me made.

Firstly the IAC Bus driver must be switched on via Applications>Utilities>Audio MIDI setup in the Mac finder window.

After this stage we can then move onto the initial program linking process. I will include a small demo tutorial in this section in order for others to follow in my footsteps.

Now open Ableton Live and load whatever set you like and apply the following steps to it

Next you have to configure the 3dMix patch in Max/MSP.
In this screen capture, I have labeled the different sections/variables you will be changing as A, B, C, D, E, F and G for reference purposes.






First, in section C, double click 'midi out b' to the right of 'curser' and select 'IAC Driver Bus 1'.Now do the same for 'midi out b' next to 'fid note' and 'fid cc'.

You might need a scaling curve for what you want to do with your table but in this instance, you won't so in section G, turn all 3 sliders to their left most position. Now in the Ableton's menu bar click Live>Preferences and hit the MIDI/Sync tab. Under the input section turn 'Track', 'Sync' and 'Remote' on for 'IAC Driver (bus 1)', and make sure all other options are off.

Now get the loops playing by clicking the play button next to 'Loops' in the Master channel to the right of the channels dialog.

Now in the Samples Dialog at the bottom you are going to be using the Transpose controller knob. If you can't see it, hit shift+tab and it should show up. The reason I am using Transpose as an example is because it makes a very significant difference to the sound so it is easy to tell if the TUIO simulator is reacting with Ableton. If you can't see the transpose knob and cant get it to show, use something else like the attack knob in a gate audio effect. Now you are going to set a fiducial in the TUIO simulator to control the Transpose knob in Ableton (or the attack knob)
To do this, first you have to set the Max patch up to accommodate for this.

In the patch, section B, set 'angle (Z axis)' to 'rotary (increment - decrement)' then set 'Rotary mode' to 'absolute (0-127)'

When testing you don’t have to use the TUIO simulator but it just makes up a lot of time if you haven’t already got a camera plugged into you computer via ReacTIVision.

Now you will be dealing with section E of the patch that controls which axis of each fiducial is packed as MIDI data and sent to Ableton.

The axis are as follows: say you have a fiducial on the table, we'll call this point zero. +X would be any value to the right of zero, horizontally. -X would be any value to the left of zero, horizontally. +Y would be any point above zero, vertically. -Y would be any point below zero, vertically. Z-axis represents rotation, so +Z is anything clockwise past zero and -Z is anything anti-clockwise past zero.

In this case you only want to track the Z-axis because you are only using the fiducial as a knob. I will be using fiducial 0 but you can use whichever you like. So in section E on the 0 column click the all the boxes except the last 2. This will put an X in each box, and therefore won't send any of the X or Y values.

The sensitivity of the fiducial is also important, if it's too sensitive you will only turn it a few degrees and the knob in Ableton will turn to its maximum value. To alter the sensitivity of Z axis (rotation) of the fiducials, use the 'Rotary' slider bar in section A of the Max patch. I found 520 is a good sensitivity for my needs.

Now in Ableton, click the MIDI button in the top right corner. This enables MIDI mapping mode, which allows you to set Ableton's controls to MIDI controllers. Having already set up the TUIO simulator as a MIDI controller using Max/MSP, You will now set fiducial 0 to the Transpose knob (or attack knob) by clicking the knob (after turning on midi mapping mode) then in Max, drop the blue square fiducial marked '0' onto the table surface in the TUIO simulator.

If you can still see Ableton running in the background, you should see a little green light flash next to the MIDI mapping button in the top right corner.
If you don't see this light flash, the simulator is not interacting with Ableton and you'll have to backtrack and see what you've done wrong (most likely, you haven't set Ableton or Max/MSP to the IAC driver).

The light flashing means that Ableton has recorded your action in the simulator and you will see 1/0 appear on the Transpose (or attack) knob. This indicates that the fiducial you dropped on the table (0) now controls Transpose (or attack). Now hit the MIDI mapping button again to turn off mapping mode and launch the new function.

Ensure your loops are playing and go to the TUIO simulator and turn fiducial 0(when it is on the table)
To turn a fiducial, right click it (or ctrl click) and drag upwards to turn clockwise (+Z) or downwards to turn anti clockwise (-Z) this should alter the transpose (or attack) knob in Ableton, and therefore drastically change the sound.
You can apply the same technique on any knob or setting in Ableton that is available in the MIDI map mode.

For example you might want to make a fiducial slide side to side and control a cross-fader. The only difference in this application is that you would set the fiducial to respond to X axis rather than Z-axis.
This is the first crucial step to set up the primary programs that control the function of the Beat Box Reactable.

The next stage would be to set up a high-resolution camera and link it to ReacTIVision. I recommend the PS3 eye camera for this as its what I have used along with many other people in previous projects.

The camera is simple to set up by plugging it in and downloading the necessary drive called Mac cam, which is totally free. This then registers the camera onto your computer and can be accessed as a usable cam in ReacTIVision.
If your using TUIO it should automatically register a fiducial when placed infront of the camera, otherwise your have to code it using the reactivision.xml file.

The general positioning of the camera is crucial to the functionality of the table. It must be placed face up and in the center position inside the box casing. This is so the fiducials can be clearly seen when placed at any point on the surface.

The main purpose of the Beat Box is to encourage users to experiment with new tangible interfaces in order to create a live performance music set. In order to do so the users are given 14 coloured cube's which are placed around the outside of the Box surface. The visuals are constantly moving on the screen in order to draw the user to the table. Once a cube is placed on the surface 1 of 84 possible samples with begin to play. 3 cubes can be placed on the surface (one of each colour) but if 2 of the same colour are placed down the first one will be cancelled out. If 1 of each colour is placed on the surface there will be 3 samples playing at once. This is where skill and the user ear will come into play as they will try to find beats which sound good together in order to create to own musical twist.

The visuals begin to replicate the beat patterns in the form of a 3D graphic equalizer, which has colours themed with the corresponding artwork. If a cube is lifted from the surface, the beat will continue to play until the same colour stop button is put on the surface. I chose to implicate it like this due to cubes possibly being knocked about and it could mess up the users beat pattern.