Below is an article I wrote for the Adobe Developer Connection (Education Category). The full article, published July 2010, can be found at: http://www.adobe.com/devnet/edu/articles/frank_garofalo.html

- - - - - - - - - - - - - - - - - - - - - - - - - - - - -

"Multi-touch" technology has generated a large amount of buzz in the past several months. Some of the buzz is about mobile/handheld devices, and some has come from broadcast media's adoption of touch devices for their programs (such as CNN's "Magic Wall," NBC's Saturday Night Live, and ESPN). This has resulted in the term "multi-touch" being used to describe a broad range of interaction types, interfaces, and devices. Most of the uses for touch interfaces that I've seen have involved a single user interface allowing only one or two touch points at a time through tapping the screen or through gestures. However, the true capabilities of a "multi-touch" interface can accommodate multiple users interacting through any number of taps or gestures on the screen at the same time.

Multi-user Multi-touch

For the purpose of this article I do not refer to "multi-touch" to describe a smart-phone or hand-held device that can recognize only one or two points of touch simultaneously, such as a Motorola DROID; I refer to larger devices like the Microsoft Surface product that can accommodate multiple users and recognize points of touch from several users simultaneously (at least three simultaneous points of touch).

The research conducted as part of my graduate studies in the Department of Computer Graphics Technology at Purdue University (www.tech.purdue.edu/cgt/) has been a qualitative examination of multi-user collaboration with multi-touch devices. The study explored the experience of users performing a common task in a shared environment (in this case the shared environment is the single display screen of the multi-touch device). For more details about this research study visit www.multiusermultitouch.com.

Digitally Replicating a Six Sigma Brainstorming Exercise

For my research, I designed and built my own multi-touch table (which is currently pending a U.S. Patent) that could showcase the interface I developed with the Adobe Flash Platform.

For the hardware, I used an open-source software package called Community Core Vision (CCV), created by the NUI Group, to serve as a liaison between the hardware components and the Adobe Flash Player. I developed an Adobe AIR application to recreate a brainstorming exercise used within Six Sigma, known as both "Affinity Diagrams" and "KJ Analysis." Typically in Affinity Diagram exercises, participants anonymously record and submit individual ideas or concepts on separate sticky notes. Then the participants sort through the sticky notes, grouping and categorizing them by moving them physically. Through this process, unforeseen relationships between the ideas can emerge. Since this activity requires participants to interact in a shared work environment in the physical world, I chose to replicate it in a multi-touch environment to explore the implications for a multi-user interface.

Exploring the Multi-user Interface

A moderator with Six Sigma Green-Belt certification and experience working with both corporations and universities served as a facilitator for the exercises performed during user testing. I selected Adobe AIR as the client since the application could run natively on a laptop computer connected to the multi-touch table, and chose the Adobe Flash Platform to capture the experience of multiple users simultaneously interacting with multi-touch interface and hardware. In the AIR application, users were able to move virtual sticky notes on the multi-touch table, and perform the same grouping and categorizing exercises that people do with physical sticky notes in an Affinity Diagram exercise.

While both the hardware and software of the multi-touch device created a few challenges for the participants on various levels, including occasional software failures, they were able to complete the exercises. That benefited not only this study, aimed at collecting qualitative data from the participants, but also the participants themselves, who were able to take the results of the Affinity Diagram exercises back to their respective organizations on the Purdue campus.

Affirming Research Assumptions

My goal was to determine whether a multi-touch interface and device can be a practical shared environment for a team of individuals to complete a common task. The participants indicated that for completing an Affinity Diagram exercise, the multi-touch device proved to be a useful tool and believe that it improved collaboration. The moderator for the Affinity Diagram exercise was surprised at the speed with which the participants in both rounds completed the tasks, despite a learning curve for them to become familiar with the multi-touch device. She remarked that she has "not done one on paper that quickly." Furthermore, participants in the study expressed appreciation for the usefulness the multi-touch device provided. It allowed them to see all the data in front of them at once, and to see most of the actions of their fellow participants.

About the author

The submission of the Adobe AIR application I developed as the user-interface for my graduate research project has been selected as a semi-finalist in the Adobe Design Achievement Awards. Below are my submission details:

• Submission Category: Non-Browser-Based Design
• Submission #: 484
• Entry title: Multi-user Collaboration
• Number of Pieces: 1

For more details about the Adobe Design Achievement Awards, visit: https://adaa.adobe.com/us/content/adaaLive

Today I gave my defense presentation to my graduate research committee and the professors signed-off on my research report and passed me! I get to graduate in August with a Master of Science in Computer Graphics Technology.

My Defense Presentation is tomorrow and I'm ready! The Defense will take place at 1pm EST in room Knoy 373. The presentation slides are done and I've reviewed them several times already.

Grad school has been an interesting and challenging journey. I'll openly admit that there were some points along the way I wasn't sure if I was going to see the end. I'm glad it's finally coming to a conclusion. I have learned a significant amount not only about my research topic and methods for conducting qualitative research studies, but also I've learned more about myself... the cliche of "when the going gets tough, the tough get going" comes to mind.

I have several "thank yous" to give out for my sincere appreciation:

• My parents and brother
• Robert Brophy
  
• Dr. James Mohler
• Prof. Rodney Vandeveer
• Prof. Terry Burton
• Prof. Gail Farnsley
• Dr. Mary Sadowski
• Julie Talz
• My Resident Assistant Staff Members at Cary Quad from the past few years
• ... and all my friends who have stood by me (and let me vent/rant at times!), you know who you are

Thanks!

Today I received edits back from the chair of my graduate studies, Dr. James Mohler. His feedback, along with the edits, was:

I have gone through your document and I think it is in pretty good shape. I have edited and you can pick up the changes I am suggesting. Most edits are grammatical errors/small things to tighten up the document. You have done a good job analyzing and summarizing the qualitative data.

So... good news! I've completed making all of his recommended changes and submitted it back to all my committee members: Dr. James Mohler, Prof. Terry Burton and Prof. Rodney Vandeveer.

The main draft of my research paper is now complete. Here is an overview:

• Content Pages: 127 (including the Bibliography)
• Content Page Word Count: 41,894 words (including section titles, not including the table of content and list of figures)
• Total Pages: 154
  

I've completed transcribing the audio recordings of the 8 1-on-1 interviews and 1 of the 2 focus group sessions. Now it's time to start the data analysis...

From all 8 interviews there were 5,846 words transcribed, but to get a visual of the most frequent spoken words from my testing participants (plus filtering out 'common English words') below is the result - thanks to Wordle:

Today marked the beginning of Round 2 of user testing with Part 1 - the Affinity Diagram exercise. From Round 1 (User Testing - Part 1) the voluntary participant number wasn't high enough so a second round of user testing was required. The testing today went very well overall. I'm glad to say that the system didn't crash, as it did during Round 1 - Part 1 (so the modification I made to the open source code seemed to resolve the issue that was occurring).

The largest challenge with the user testing today seemed to be when the digital sticky notes would either move to the default top left corner (the 0,0 coordinate) or it would move to the location of the last touch point (which was usually another testing participants touch point). This seems to be an issue with Community Core Vision (CCV), the open source code, and the ActionScript 3.0 class which communication with CCV.

In a letter Steve Jobs released discussing Apple's "Flash distaste" (http://news.cnet.com/8301-30685_3-20003742-264.html), he states the following:

Flash was designed for PCs using mice, not for touch screens using fingers. For example, many Flash websites rely on "rollovers", which pop up menus or other elements when the mouse arrow hovers over a specific spot. Apple's revolutionary multi-touch interface doesn't use a mouse, and there is no concept of a rollover. Most Flash websites will need to be rewritten to support touch-based devices. If developers need to rewrite their Flash websites, why not use modern technologies like HTML5, CSS and JavaScript?

Now as a designer / developer for multi-touch interfaces with the Adobe Flash Platform, I may be a bit biased here, but I disagree with his statement on several points. Most technologies were designed during the PC age, including Flash, the Web, HTML, and even several products from Apple. The mouse at the time was the most cost effective input device to replicate actually clicking and selecting items on a computer screen/display. Now with the continuing evolution of multi-touch devices, there are several new interface advantages and disadvantages emerging.

"Rollovers" can still occur in a multi-touch interface to help indicate to the user what element of the UI the user has selected. While I do admit, with multi-touch there is a blurred line between a "rollover" and a "single-tag/touch," so depending on the interface these may need to be coded differently to acknowledge a touch vs a mouse click.

As for the "modern technologies like HTML5" - we've seen time and time again with various Internet browsers slightly different implementations of web standards created by the W3C and I doubt HTML5 (http://dev.w3.org/html5/spec/Overview.html) will be any different. The challenge with this to web designers / developers is creating a consistent user-experience across all Internet browsers and computer operating systems. At least with Flash, a consistent, engaging experience can be achieved.

The digital post-it notes being sorted included various popular social networking sites, such as Facebook, YouTube, Flickr, Twitter, etc. Special thanks for volunteering to be video taped - Sean S., Brian E., Ben T., and Mike M.

On April 23rd 2010 a provisional patent application was filed with the United States Patent and Trademark Office (USPTO). The provisional number is 61/327,354 with Frank J. Garofalo and James L. Mohler listed as the inventors, and the Purdue Research Foundation (http://www.prf.org/otc/) as the assignee.

Yesterday (April 19th 2010), I started part 1 of the user testing for my research project regarding the multi-touch device and software application I've been developing. The software application is based on the Six Sigma Affinity Diagram (aka KJ Analysis) exercise. This is typically used to sort through qualitative data (such as ideas from a brainstorming session) to find unforeseen relationships between the the data items / ideas by grouping them into categories.

I was granted permission by Dr. Mary Sadowski, Associate Dean for the College of Technology at Purdue University, to ask faculty members from a committee within the College of Technology to use my multi-touch device and software application to aide them in sorting through some ideas/outcomes regarding a project they are working on.


Image 1 - Doing a run-through prior to user testing during set-up.


Image 2 - User Testing.

During the user testing the software application unfortunately crashed twice. Each time the users had to restart the Affinity Diagram from the beginning. After the second crash, I went into the code and quickly made some alterations. The problem was occurring with an open-source class written in ActionScript 3.0 which processes data received from Community Core Vision (the open-source software which interprets the images received from the camera component of the device's hardware). I added some "try / catch" statements. This seemed to resolve the issue since the software application didn't crash after the modifications.

Overall, the user testing proved to be successful. Excellent data was collected both regarding the interaction of user-to-user and user-to-interface. Also, notes for improvements to the software application.

One of my friends shared this video/article link with me. The pen looks easy to make - basically a stylus with an infrared light on the end. As for the software - it's very impressive. I look forward to experimenting with the other multi-touch apps I can come up with once I'm done with the app I'm creating for my grad research.


Source: http://www.istartedsomething.com/20100409/microsoft-research-manual-deskterity-synergy-pen-touch/

I originally hoped to use the ColorPicker component in Flash but I was having too many problems trying to modify the code behind the component to get it to recognize the Touch Events from CCV and TUIO. For the sake of time and meeting my approaching deadline, I decided to ditch that idea and just make a set number of colors that will automatically be selected as groups/categories are created.

Below is an example of what I've come up with so far.



Still more to come, next is the ability to vote on the priority of categories/groups.

I've made progress on my multi-touch software. I can now have multiple assets displayed on the device being touched (selected) at the same time and have events respond respectively to each individual asset selected. Getting this to work was a major accomplishment. Thank goodness for error catching.



Now I've run into another problem trying to save over an existing XML file with updated data. Spent about 3 hours this evening trying to figure it out without much success. I've searched Google to see what other people have tried but I've yet to find a working solution... slightly frustrating. If I find one, I'll post about it.

So I've been working with the ActionScript class RotatableScalable (which can be found on the NUI Group web site at: http://wiki.nuigroup.com/Building_Your_First_Multi-Touch_Application_in_Flash). However, I found a major flaw with that class that occurs when rotating an asset object that extends the RotatableScalable class. The object is rotated around its 0,0 coordinate (or in other words the top left corner of the asset). From preliminary testing, this proved to be very disorientating to users since the asset would spin away from the users' fingers.

Chris Yanc created a revised version of the RotatableScalable class called RotatableScalableRev which resolves that issue. For more details about his version, visit: http://www.cyancdesign.com/2009/05/understanding-rotatablescalable/

I'm so glad to get this working correctly now - thanks Chris! :D

So now that the hardware of the multi-touch device is essentially done (there are some things I could probably do to further improve it), I've moved on to the software development side. I've been working in Adobe Flash CS4 with ActionScript 3.0. It's been a lot of trial and error testing code as I go. However, I'm making progress - just slow progress.

One of the major hurtles was a problem I ran into with embedFont = true; making the dynamic textfields disappear. But I found a solution and blogged about it - link.

SixSense Technology (link):

The multi-touch device I've been building since June is finally working now. There can be some tweaking and calibrating to improve its performance, but at least now its working.



Above pictures are using demo apps from the NUI Group.

Received the materials to build my multi-touch stand this past Thursday. There were a few minor issues with the materials and design, so now I need to wait for the supplier to me some fabricated pieces. It's a little frustrating right now since it's so close. I can't wait to get the hardware components to a completed stage (...aka satisficing) so I can focus solely on the software ui with Adobe Flash.

My sincerest thanks to Mark Miller from Miller & Associates. I met Mark at the Adobe MAX 2009 conference in Los Angeles, California. His company had teamed up with Adobe to build a multi-touch device for the conference to showcase using Adobe Flash for multi-touch devices. He was generous enough to provide me with a sheet of enlightened acrylic outfitted with an aluminum track and infrared LEDs. At Adobe MAX you truely can "connect, discover, and inspire." Here are some pictures of the enlightened acrylic with CCV running:




Thanks again Mark for supporting a graduate student.

Today I spent time working on AS3 code for my multi-touch device. I found the following links very helpful:

• http://wiki.nuigroup.com/Building_Your_First_Application
• http://code.google.com/p/multitouchas3experiments/

A few days ago I also changed the configuration of my mutli-touch device from a tabletop (horizontal) orientation to a wall (vertical) orientation. I mainly did this after the PS3 Eye camera arrived to see if the PS3 Eye would detect the hot spot created on the acrylic from the projector's light beam. I was pleased to see that the camera didn't detect the hot spot.

My new PS3 Eye camera arrived to replace the other infrared camera I had been using. The PS3 Eye was modified to capture infrared light. After swapping out the camera, my multi-touch device was responding much better.

Some progress was made with the multi-touch device I've been building. Using the CCV software and some demo Flash apps provided on the CCV web site, here are some pictures and a video:

I've found, in addition to the NUI Group Forum (www.nuigroup.com), the Touch Factors Blog to be a very helpful resource: http://www.touchfactors.com/?p=103

I decided to try to remove the silicone layer on the acrylic. I purchased a plastic scrapper, similar to that used for removing decals from vehicles. The instructions for the scrapper stated to use water or a solvent. I tried both and only was able to get up small fragments. After talking to the maintenance staff at my work location, they said that the silicone has most likely set on the acrylic and its not going to come off easily without damaging the acrylic (which the acrylic would need to remain unscratched and transparent).



I gave up the idea of removing the silicone and decided to purchase a new piece of acrylic.

I think there is a possible issue with the silicone layer of my multi-touch device. I had tried to do the method described by Tickerman on the NUI Group Forum (http://nuigroup.com/forums/viewthread/2197/P0/; YouTube Video: http://www.youtube.com/watch?v=IEgoVBfF68Y&feature=player_embedded). Tickerman calls it the "siliconed vellum" method.

I was unable to get the silicone completely smooth and it seems that the infrared camera I have is having difficulty distinguishing the blobs created by fingers in contact with the mt surface and the impurities/imperfections of the silicone.


Screen capture of CCV without contact to the MT surface


Screen capture of CCV with five fingers in contact on the MT surface

I've started building my own multi-touch device after reading several posts of blogs and forums through the NUI Group web site (www.nuigroup.com). I complied a list of commonly mentioned components and ordered them on-line. If anyone wants the list I can post that on here.




Yesterday I tried to run my multi-touch device with the Community Core Vision (formerly tBeta) software from the NUI Group. However the software wasn't detecting the blobs created by the fingers in contact with the surface of the device. It seemed that it was picking up too much heat reflecting off the backside of the acrylic/Plexiglas. I also tried to use a mirror to bounce the projector's image to the backside of the multi-touch device with the infrared camera next to the mirror to see if this would help with the situation. However, there wasn't a noticeable difference. Back to the grindstone to figure out what's wrong and why it isn't working correctly.

Today I had an opportunity to meet with professors in Purdue's Engineering Education department who have a MS Surface. Spent about 3 hours meeting with them. It was great to play with a multi-touch device that was professionally manufactured.

I was surprised that the multi-touch capabilities are only available when in user mode which seemed to run a Silverlight app that serves as a core menu to launch other apps. Already having previous knowledge, from researching, about gestures for multi-touch devices I still found the gestures very natural and intuitive. When I was using the device by myself, I found the responsiveness of the device and applications to be acceptably accurate. There were a few apps, such as the MSNBC News Sphere to be difficult to turn the sphere. During this time, we at most had three individuals interacting with the Surface and the responsiveness was good.

Our goal is to test using Adobe Flash and Adobe AIR on the device utilizing the t-Beta protocol (found on the NUI Group web site: www.nuigroup.com). Since the device is not ours, we are waiting for permission to install the t-Beta program on the device. I'm excited to see how the device will respond to a non-Microsoft platform. There will be more blog posts in the future once we move forward with testing Flash on the MS Surface.