Large-scale multi-touch displays are widely used in real-world practices to support various collaborative works [15,16,27,36]. These displays detect concurrent multi-user touch inputs on a large space for information exploration and organization. They are targeted for workplaces that involve design and analysis activities [13,17,21,26,41]. In these contexts, the multi-user touch displays serve as platforms to facilitate creative activities, such as information exploration, idea generation and exchange, and decision-making.

Prior studies on contextual use of multi-user touch displays provide an understanding of interaction and collaboration styles. Two categories of interaction—simultaneous and sequential—are differentiated during collaborative use of virtual spaces [18,20,24,27,31,34]. Insimultaneous interaction, more than one person touches the display simultaneously to manipulate digital objects—for tasks that can be done in parallel [27], or for collaborative tasks that require multiple users to complete [24]. In contrast,sequential interactions are guided by social protocols in which people negotiate and take turns using the device [28].

Clearly, simultaneous and sequential interactions afford certain tradeoffs. Simultaneous interactions support the distribution of tasks among people, opening possibilities to finish the task more quickly. But this task distribution may weaken awareness of others’ actions. Sequential interaction can enhance awareness and build a sense of group cohesion by encouraging collaborators to observe and participate in actions, though it is unclear whether this sequentiality is the best use of time device interaction capability. Investigating factors that influence people’s decisions to work simultaneously or sequentially—and roles and impacts of technology in encouraging their behaviors—will broaden our collective knowledge about ways in which technology can support collaborative efforts.

To better understand simultaneous and sequential interaction in creative collaboration with multi-user touch tabletops, we conducted an observational study of pairs and triads completing an ideation task on a 55” tabletop touch display. Ten design sessions are observed to investigate the patterns of the collaborative interaction. Four collaboration factors gleaned from prior research are investigated as factors in collaborative idea synthesis:coupling style,interaction stage,territoriality, andsemantic actions. The four factors reflect workspace awareness knowledge [10] identified by Gutwin et al. and provide a platform for understanding how designers engage in collaborative tasks. Our research seeks to learn how users collaborate to foster future design of software and technology leveraging large multi-touch displays. The contributions of this work can be summarized as follows:

• Present four collaborative factors that influence simultaneous and sequential use of tabletop displays.

• Explain when and how interaction style influences collaborative factors.

• Identify how knowledge of simultaneous and sequential styles drawn from each collaborative factor provide recommendations for system design.

The number of detected touch points technologically limited early studies on co-located collaboration with digital display tools. Single-touch devices based on fingertip or stylus input best support turn taking, not simultaneous collaboration [2,35]. Thus, research on collaboration with these devices focused on social protocols for turn taking [32]. Advances in touch and gesture recognition technologies enable support for simultaneous interaction, raising questions about when and how to use it.

Both simultaneous and sequential interactions with large-scale multi-touch have been explored previously. The simultaneous and sequential nature of interaction techniques can be used to improve collaboration. For example, studies show that the walk-up-and-use nature of a public multi-touch display consists of both simultaneous and sequential interaction [15,22,27]. Information analysis and search tools on multi-user displays facilitate collaborative sense-making by supporting simultaneous and sequential use of the digital materials [18,25]. Map and animation planning enforce simultaneous operations with gestures that have to be performed together to enhance group awareness [7,24]. Interaction techniques such as virtual hands or arms are studied in terms of influences to the simultaneous and sequential interactions [4].

We introduced four factors in the introduction that have been used in prior work to observe and explain collaboration phenomena. Gutwin et al. explored workspace awareness and suggested five questions, withwho,where,what,when andhow as basic elements for workspace information [10] to guide identification of factors in a study. The four factors cover social, time, space and interaction aspects of co-located collaboration, giving a comprehensive examination of simultaneous and sequential interaction on tabletop displays (Table 1).Coupling styles capture how collaborators engage in mixed-focus collaboration, when people switch between individual and shared works [38]. It reflects the degree people are aware of others’ presence and identity, as a part of thewho question. Modes ofloose- andclose-coupled collaborations reflect social aspects of collaboration [10,20,21,35,42]. Understanding coupling styles is useful for understanding co-located groupware.Interaction stages describe temporal change in collaborative behaviors toward finishing an ideation task. Tabletop systems provide flexible ways for transitions between activities [35]. Collaborative design as a reflective and evolving process is generally described in terms of multiple steps or stages [33,39]. In each, the nature of different activities influencewhen people collaborate with others and interact with digital artifacts [29,33,39].Territoriality spatially probes how collaborators divide shared interaction space to organize ideas [11]. Inpersonal space, individual ideas are engendered, whileshared space is for idea exchange and integration [34]. Control and availability of digital objects in territories influences the arrangement of users and how individuals engage in personal and shared tasks [34,35]. As an example of location and reaching issues, territoriality reflects information fromwhere question.Semantic actions, concerningwhat actions are performed onwhat artifacts, bridge user cognition and interactive materials. Studies on interaction patterns with social meanings have evolved a gestural vocabulary with which users communicate with design objects [5,30,31]. These actions serve two purposes. Some gestures resemble interactions with physical objects and assist users in indicating relationships or similarities [8,26]. Other gestures have social meanings; e.g., two corner-shaped hands which block other’s view can convey privacy [44], and co-dependent gestures enhance awareness of joint actions [24]. Semantic actions influence interaction with shared materials and further coordinate simultaneous and turn-taking activities [35].

Technological advances, and accompanying tools, raise questions of when, where, and how simultaneous and sequential interaction are used effectively in collaboration. To further the knowledge on the collaborative interaction with multi-user touch display, this paper examines simultaneous and sequential interaction in a common goal-driven, free-form interaction ideation scenario using digital cards on a multi-user touch display. Multi-user multi-touch displays are increasingly used in design-related collaboration [9,14,21,41]. Design tasks require participants to cooperate with each other in using digital materials, toward managing task progress and making decisions. Ideation tasks, seen in collaborative use of tabletop displays, involve activities like information gathering and decision-making [6,7,40].

Simultaneous and sequential interactions play different roles in the collaborative interaction with multi-user touch displays. Simultaneous input techniques support collaboration by giving each person equal opportunity for information access, and, accordingly, enhancing equality of participation across the group [16] and increasing cooperation [24]. In addition, when the interaction task consists of distributable work, simultaneous interactions allow collaborators to work on different parts of the task in parallel. However, simultaneous interaction may also result in interaction errors. Sequential input coordinated by social protocols is another collaboration approach with multi-user displays. Benefits lie in avoiding conflicts from simultaneous interaction [15], and sequential interaction also can enhance group awareness by leaving users sufficient time to observe each other’s’ activities [16]. But requirements of sequential use by a system or method might reduce efficiency by cutting down on opportunities for parallel activities [22,27]. To provide a better understanding of the two interaction styles with the multi-user touch display, we focus on an ideation task to probe the patterns of collaborative activities in the four examined factors.

We investigate two interaction styles through four collaboration factors in tabletop workspace. This section details the identification of these factors and discuss the sub-categories in each category (Table 1).

4.1Coupling Style

A group’s collaborative coupling style describes the social manner in which individual collaborator participates the work. In collaborating through a multi-user display, some tasks benefit from joint attention and/or the joint action to complete, while other tasks can be done in parallel. Focused attention involves more verbal communications, usually centered on one or more digital objects. With distributed attention, communication is more occasional, and manipulations by different collaborators are not necessarily related. The group transits between focused and distributed attention, influencing manner of interaction with the display. We examine two types of coupling styles,loosely- andclose-coupled collaboration [11,18,20,38]. Communication and attention are the main indicators to distinguish coupling style. In close-coupled collaboration, the group has active discussions that focus on the same digital objects. In loosely-coupled collaboration, each group member focuses on objects in parallel without significant communication. Observing sequential and simultaneous interaction within each style will help understand how attention and conversation factors influence collaborative interaction.

4.2Interaction Stage

This paper leverages three temporal stages of interaction which outline the design process, drawing primarily from the PIC-UP card set and the design tool [43], but prevalent across much of the design activities. We usebrowsing,organization, andspecification as three interaction stages to observe the temporal changes of simultaneous and sequential interaction.

Browsing Stage is the initial step in using the interface to understand content in working space. When browsing, the group may not have a clear idea about the tool or task. Browsing gives users a perspective that helps to explore ways to use the system and understand the information provided by digital objects. Interactions when browsing are tentative and less coordinated, since attention and action of different users generally focus on their own interests. Parallel work helps users learn information and quickly identify ideas.

Organization Stage is associated with the arrangement of design materials toward deeper assessment of ideas from the browsing stage. The evolution of ideas from browsing to organization lie in relationships identified among digital elements. The group exchanges ideas and tries different combinations of the elements, to collaboratively make decisions on which elements to use. The shared display at this stage serves as a space to exchange the idea, so the interactions usually incorporate sharing or evaluating digital objects. These types of activities usually attract group attention, and the joint attention influences the manner of collaborative interaction.

Specification Stage involves refinements of the structure of the digital objects that comprise the design. As ideas are focused, details of digital elements and relationships among them are made more definite and clear. Activities in this stage include arranging elements and specifying relationships and interactions. Compared to organization, the arranging of the digital elements within this stage are no longer an attempt to find possibilities; they represent more certain and finalized modifications. As such, most interactions at this stage happen to nearly finished design through sequential access that help collaborators avoid mutual interference.

4.3Territoriality

It has been widely observed that collaborators createpersonal territories andgroup territories in interacting with multi-user displays [4,34,41]. Personal territory is an area where an individual participant obtains digital resources and explores design alternatives. This area leaves collaborators a private space to clarify personal ideas before introducing them to the group. Group territory is a shared space to assemble ideas from each individual that serves as a space in which the group exchanges ideas and resources. The segmentation of the interaction space influences the manner of interaction. To further explore how territory influences the simultaneous and sequential interactions, we observe the two modes of collaboration within two types of territory. Personal territory interactions include activities within personal space, namely interactions to organize individual idea, or move interesting cards into the personal space. Group territory interactions consist of activities that happen in the group space, which are the modifications made to the ongoing design.

4.4Semantic Actions

From creating a diagram to organizing data files with a multi-user touch display, collaborators need to annotate, sort, group, link and delete digital objects to make sense of the materials [8,15,16,26]. This paper examines 5 semantic actions with the multi-user display—emphasizing, sorting, grouping, linking and deleting—gleaned from prior studies with interactive information synthesization that cover most design-specified actions [23,26,30]. When a user observes the object set and the elements are especially inspiring, the user mayemphasize that object and share it with teammates with actions that include moving the object, pointing at it, and elaborating on an idea. After the user collects ideas, it is necessary tosort the digital objects and select suitable ones, perhaps bygrouping similar or relevant information. By clustering digital objects, an idea group is used as an integrated entity to further connect to other materials. For key connections, collaboratorslink the feature pair by drawing lines between digital objects, or by positioning them closely. During organization of elements, if digital objects appear to be less important or relevant, usersdelete these elements through gestures such as moving them to the corner of the display.

This section describes our observational study of simultaneous interaction and sequential interaction in a small group design activity. Leveraging a card design system implemented on a multi-user tabletop display, 10 small groups of 2 or 3 collaborators created designs with digital materials. We chose a design task because it reflects the need to experience different coupling styles, to shift between creative design stages, to manage space utility, and to perform gestural actions, thereby fulfilling the goals of our observation.

The observational experiment was conducted with 19 undergraduate students between 18 and 30 years of age. To ensure the participants have basic design knowledge, participation was only open to students from two upper-level design-related classes: Human Computer Interaction (N = 16) and Information Design (N = 3). Students from the design classes are familiar with the nature of the task, which helps curtail the learning process and avoids confusion on task requirements. Participants worked in a team of 2 or 3 people, and they could take part in the study either once or twice. To help ensure diversity of ideas in the design process, no two participants could collaborate in a group more than once.

5.1Multi-touch Supported Design Tool

The card-based multi-touch supported design tool used in this study is a design application [12] running on the Microsoft Perceptive Pixel (see Fig. 1). The Microsoft Perceptive Pixel is a 55-inch multi-touch display enabling simultaneous finger operations of different users like touching, pinching and dragging. The design application running on it consists of a blank working area in the center and design cards along each side (Fig. 3). Each design card can be moved, rotated and resized by different designers simultaneously with the fingertips. A toolbox, located on each side of the table, provides a drawing tools and a commenting tool. Using a pencil tool, users can draw lines of different colors and thicknesses with the finger. An eraser tool and an undo tool are provided for correction. A commenting tool creates a textbox and a virtual keyboard for editing the typed comments. Each textbox can also be moved, rotated and resized.

Using the collaborative design tool on a Microsoft Perceptive Pixel system.

Full size image

5.2Collaborative Task and Digital Card Creation

The design task is to design a novel technology solution to help 5–7 year old children manage their emotions, such as anger and anxiety. The digital cards used in this study are created based on three workplace factors—social context, activities and artifacts—as described in the design requirements. Accordingly, we designed three types of cards corresponding to each of the dimensions: problem cards, activity cards and technology cards (Fig. 2). The 30 problem cards include emotional problems summarized from the Preschool Anxiety Scale (PAS) [37]. The problem cards aim to provide hints on the context of the design and help the designers understand the emotional problems of the children. 31 activity cards show common activities like reading, walking and listening to music with which the children can alleviate negative emotions. The technology cards have 37 popular digital devices like tablet, smartphone, and Google Glass.

Examples of the three types of design cards. From left: problem card, activity card, technology card.

Full size image

5.3Process

19 participants formed 5 groups of 3 members and 5 groups of 2 members. Each group used the design system to complete the design task. First, the design task was described to the group. The method to manipulate the design card and draw the lines was demonstrated to the participants. Each participant then tried the system. After the participants became familiar with the tool, the group was informed that the design outcome should be presented in a form of design poster, and an example poster was shown and explained. An overhead camera looking the tabletop display (Fig. 3) recorded video of the entire design process. After finishing the design poster, each participant completed a 6-item questionnaire probing aspects of the experiment: using the interface, exploring the digital cards, organizing ideas, completing the design, collaborating with teammates, and general feedback. The questions include an open-ended writing portion to let participants leave their comments on each aspect.

Video screenshot of the investigation.

Full size image

During recruitment, we allowed participants to take part multiple times in the study. 6 students participated the study twice and the rest participated once. Repeated participation sought to explore whether participating multiple times would lead to dominance and other changes in coupling styles, but the data and questionnaire shows that participants engage similarly each time, suggesting social conventions still guide the manner of collaboration [1].

5.4Coding Method

To examine the relationships between the four collaboration factors and the simultaneous and sequential interaction, we first coded the video. Simultaneous and sequential interactions are determined by whether the participants touch the display at the same time. If two or more participants touch the display at the same time, that interaction period is tagged as simultaneous, otherwise as sequential.

The video clip was first divided into 3 parts according to 3 interaction stages. In the browsing stage of card-based groupware, the main task is observing the card set and discussing ideas over single or a small subset of the cards. We define the browsing stage as the period during which the most of, if not all, cards are browsed, ideas are brainstormed and useful or relevant design cards are selected. We use the time after which no cards being observed for the first time as the end of the browsing stage. The organization stage is a process of evaluating the initial ideas, make selection, and arrange the desired cards. A reversed observation method is applied to dissect this stage: the design result is referenced first, and second is the time when the final design plan is used, thus separating the organization stage and the specification stage.

We then used an event-based method to code the other 3 factors. Two research investigators reviewed the entire video separately. If the status of any factor changed, the investigator recorded the time stamp and the statuses of coupling style, territoriality and semantic actions according to the definition described above. For the coupling styles, if the participants communicate with each other (or keep silent) more than a short while (2–3 s), we consider them to be switched to close-coupled style (or loosely-coupled style, respectively). For territoriality, if a card is being or has been shown to all participants, and no user moves it back to his side, we consider this card to be in group territory. Interactions with shared cards are categorized as group territory activities. Otherwise the card is considered as personal card and interaction with it is categorized as personal territory activity. As mentioned previously, four semantic actions are coded based on the purpose of the interaction.

Emphasizing actions include pointing, scaling, or mentioning a single card to explain an idea.Sorting actions compare and select among cards and pick useful ones.Grouping actions move several cards together to form a card cluster.Linking actions align two cards together or draw lines between cards to show relationships.Deleting include actions which move the cards from the center to the corner.

Comments from the post-experiment questionnaire were analyzed with affinity diagramming. Comments were extracted from the questionnaire and placed on note card (181 cards in total), with one focused point on each card. 3 researchers, all with affinity diagramming experience, sorted the cards as a group, clustering and labeling related sets of cards as they emerged. The clusters of feedback are discussed in relation to the core themes of this paper.

All groups successfully finished the design tasks and created a design poster, taking between 17 and 55 min. Average time taken was 33.12 min (SD = 11.26). Collectively, design groups showed good understanding of the content on the digital cards, with only two groups inquiring about two of the technology cards during the design period (about the MYO armband and the FuelBand). There were no major usability issues that interrupted the collaborations. Though several participants perform the task twice, they still followed similar progresses in the ideation task, suggesting that the influence did not significantly impact the results.

Overall, participants far more frequently interacted with the display sequentially than simultaneously. 10 groups spent an average of 1474.10 s (SD = 564.53) on sequential interaction and 513.10 s (SD = 214.84) on simultaneous interaction. The results revealed that the time on sequential interaction is longer than on simultaneous interaction.

Despite knowing that they can interact with the device simultaneously, the tabletop is used more in a turn-taking manner [31]. A typical design procedure is that the participant teams spent some time placing the cards around the table for better visibility, and spent the majority of the time discussing the cards, selecting useful elements, and taking turns organizing the cards. Results of each category are presented in Fig. 4.

Time of sequential and simultaneous interaction for four observed factors.

Full size image

6.1Interaction Modes with Coupling Styles

On average, the participant groups spent 24.08 min (SD = 12.67) designing closely together and 9.04 min (SD = 6.92) working independently. When focusing on the same set of digital objects (close-coupled), groups spent 1237.2 s (SD = 647.02) turn-taking with the table. The average time for simultaneous touch during close-coupled collaboration is 207.60 s (SD = 144.86), smaller than sequential. When groups focus on different objects and work in parallel (loosely-coupled), average times are 236.90 s (SD = 270.02) for sequential and 305.50 s (SD = 196.59) for simultaneous interaction.

In closed-coupled collaboration, collaborators often have tense communication and pay attention to the same design issue. These conversations are usually about suggesting and selecting ideas with the cards. Therefore it benefits participants to keep aware of each other’s thoughts and behaviors to reach common ground. Compared to simultaneous interaction, sequential interaction results in more time to observe others’ activity and evaluate ideas. Since simultaneous interaction with shared materials also might bring interference to conversation and action, participants generally apply sequential interaction to avoid potential conflicts. In loosely-coupled collaboration, participants deliberate and modify different parts of the design. Less mutual influence reduces the chance of conflict; therefore participants can use the card at the same time. We noticed that in loosely-coupled collaboration the sequential interaction time has similar range with the simultaneous interaction. From our observation, participants sometimes look at and think about different cards in parallel without actively moving the cards. The sequentially in this case is due to coincidental touch.

Results of the post-experiment questionnaire also imply that participants appreciate being able to access the table simultaneously, but still desire a close partnership found in sequential turn-taking. Regarding simultaneous access of the table, a participant in Group 6 (G6) noted“I thought that the interface had a lot of options and free space which made me feel like I could do whatever I wanted.” A participant in G9 wrote“I like how we were all able to stand up around the table and interact simultaneously on a single platform rather than individual screens.” The openness and equality of using the design table benefits the collaboration, since simultaneous use of the cards gives the participants room to think and work individually, while not blocking the channel for communication.

We also note a common concern on how cards scaffold idea exchange. For example, a participant of G1 commented on the system“hard for some people to see cards of the people sitting opposite to them.” A G4 participant mentioned“sharing cards with other group members is sometimes tedious due to having to rotate and re-rotate cards.” These concerns reflect that participants wish to know what others are doing and when they are willing to share ideas. These activities mirror the turn-taking observed in card processing. Though there could be a better method to share cards, the close-coupled collaboration helps the team share ideas and maintain mutual understanding of the task.

6.2Interaction Modes in Three Stages

The average time span of the browsing, organization and specification design stages are 897.60 (SD = 390.58), 476.00 (SD = 286.71) and 613.60 (SD = 373.13) s. In the browsing stage, the times for sequential and simultaneous interaction have relatively close mean values of 499.30 s (SD = 297.95) and 398.30 s (SD = 145.37). However, in the organization stage, the average times for sequential and simultaneous interaction are 392.70 s (SD = 279.98) and 83.30 s (SD = 86.29), with the former higher than the latter. Moreover, in the specification stage, sequential interaction time (m = 582.10 s, SD = 351.41) is longer than simultaneous interaction time (m = 31.50 s, SD = 47.07).

The simultaneous interaction mostly happens at the browsing stage. This finding furthers the observation in [31] that simultaneous interactions were conducted merely at the beginning of the task. The exploration of the digital materials is the preparing phase through which collaborators become familiar with the design materials and collect initial ideas. Participants need time to browse the cards in parallel to clarify individual ideas, which leads to the simultaneous use of the cards. But when interesting ideas are identified, participants need time to briefly communicate and evaluate the idea; therefore sequential interaction also happens for similar time. In the organization stage, participants take turns proposing and evaluating ideas. The participant who suggests an idea usually takes control of the cards, and passes control of the cards to the next solver when finished. The specification stage has a clearer common goal in design. The changes and decisions made to the cards are important and definite. Therefore the simultaneous interaction in this stage is even less—collaborators need to pay attention when an idea is finalized; unexpected moves will distract the team and interfere with decision-making.

In completing the design task, interactions with cards reflect participants’ thoughts on design. A participant (G5) commented on the browsing stage“I felt cards on the table helped throw out a bunch of ideas, and the layout/tools along with the cards helped create an environment to nurture creativity and outside the box thinking.” Another participant (G6) noted during idea organization“I was inspired to think of new ideas, by seeing all of the possible problems and technology that could detect such problem at once. It allowed me to quickly decide on a path to take.” But the number of the cards sometimes highlights problems with idea organization, since“with all the cards however, sometimes the screen was a little bit chaotic and we would lose track of some of the cards” (G10). Seeing and using the cards is a process of clarifying ideas [33,39]. The phases of idea generation, synthesization, and specification reflect how participants use the design elements. Collecting possible ideas tends to be done in parallel, so cards are touched simultaneously in the browsing stage. Organizing and specifying relationships between ideas need joint attention, so card manipulations happen more in a turn-taking manner.

6.3Interaction Modes in Different Territories

For the interaction in different territories, the average time participants spent in personal territory and group territory are 1584.60 (SD = 521.38) and 402.6 (SD = 197.89) s. The average time for sequential interaction and simultaneous interaction are 1417.90 (SD = 514.39) and 116.70 (SD = 112.57) s in group territory, with the former longer than the latter. When focusing on digital objects in personal territory, the average time is 56.20 (SD = 72.29) s for sequential interaction and 346.40 (SD = 167.53) s for simultaneous interaction. The simultaneous interaction in personal territory is longer than sequential interaction.

In personal territory, participants have free access to the cards and usually interact with them without concern for other participants. The simultaneous interaction in personal territory reduces the time to finish the parallel. Being able to work independently on different cards helps designers generate rich and diverse ideas. Another case of simultaneous use in personal territory is removing the less useful cards. Considering the number of the cards, the affordance of removing the card simultaneously reduces the time to narrow down the design options. The cards in group territory are shared by all members. The co-ownership of the design materials makes individual participant tend to acquire other’s attention or consent before making changes. When working simultaneously in the group territory without proper awareness of each other’s action, errors might be caused due to the conflict of operations.

Regarding transitions between two territories, the dragging and dropping feature of the digital cards gives the participants an easy way to integrate ideas. “Integrating and changing our ideas was very simple, as we just had to drag cards around” (G6) and“it is easy to add/remove ideas and integrate them to the poster by moving the cards … we can have a better view of all the technologies we want to use, so that we won’t miss/forget something during the process.” (G8) However, some other features hinder free use of personal space. Some participants noted that the tool’s moving and drawing modes block simultaneous use of the table:“However, when drawing or commenting, it put the whole table in that mode (drawing mode) and the other person had to just watch. It would be great if one person could comment or draw, and the other people could still do other activities” (G10) and“if the group started working with one tool, and idea requiring the previous tool may be put on hold.” (G8) Participants wish to have the personal space always available even when others occupy the shared space to help quickly record ideas.

6.4Interaction Modes with Semantic Actions

Five semantic actions are observed with different frequency of use. Linking of cards takes the longest time (m = 894.1 s, SD = 363.81), and deleting is the least performed action (m = 110.6 s, SD = 133.51). The average time for emphasizing, sorting, grouping are 303.6 (SD = 257.72), 502.6 (SD = 298.88), 176.3 (SD = 122.8) s. The five actions also show differences in terms of two modes of interaction. The result indicates that emphasizing and linking are performed more in turn-taking mode than simultaneous mode. Grouping and deleting, which are the two least used actions, show no significant differences with interaction mode. For the sorting, we noticed that average time in simultaneous interaction seems higher than that in sequential interaction. This is perhaps due to that group 7 spent nearly 15 min taking turns to evaluate and sort every one of the 98 design cards.

Our observation shows that different semantic actions are conducted with different interaction modes. Emphasizing one card denotes the discovery of an interesting idea. Linking a pair of cards suggests designers identify or confirm an important relationship. These two actions usually involve pivotal ideas or decisions that inspire the team and push forward the design. As such they ask for joint attention and consensus from other members in the group. The necessity of group awareness makes the emphasizing and linking actions being conducted in a sequential manner. Different from emphasizing and linking, card sorting means ranking several cards based on some user-defined criteria. From our observation, most group members pick their own interesting cards and make the rest of the design grounded on the selected cards. So the sorting process is generally conducted in parallel in which participants can locate cards of individual interest very quickly. We also note that group 7 discusses and evaluates every card to decide which cards to use. The sequential reading and discussing of the cards increases the time to sort the ideas. Grouping and deleting are used less than the former three actions and are conducted both simultaneously and sequentially. Grouping was used to complement an idea by adding one or more cards to an existed card cluster. On the contrary, deleting happens when cards added previously appear to be less useful or irrelevant as the design evolves. In our study, the simultaneous adding or removing of cards usually happens when the target card does not impact the current design very much, while the sequential interplay of these actions usually involves cards of certain importance.

One noteworthy technique to emphasize and sort digital cards is enlarging or shrinking card, because “the resizing ability makes it easy to assign important/priority for each card” (G9) and“being able to resize them gave us the ability to make certain cards appear as more important than other, which may be a desired ability in designing.” (G7) The resizable digital cards offer a unique affordance in contrast to paper cards: different sizes represent different importance, and the better visibility makes key elements easy to capture. Also, we observed that when a participant zoom the card to take up more screen space, the intentional or unintentional intrusion draws the group’s attention, triggering a shift from simultaneous exploration to a turn-taking conversation.

The simultaneous dragging and dropping expedite creating card groups to cluster similar elements. The participants agreed that “it was very easy to share the cards or create groupings by dragging” (G3) and“it was incredibly easy to create groupings, by simply dragging cards to their desired group.” (G6) But when the card group grows large,“it would be great if there was a way to select cards and make them into a group, whenever you wanted to move a group of cards that you had related, you had to do it one by one”. (G10) These comments suggest that the participants adopt the provided touch interaction intuitively, but still wish the system to support new semantic actions for manipulating a group of digital artifacts.

To expand the understanding of simultaneous and sequential interaction with multi-user touch display in creative design, our observational study examined interaction within a goal-oriented design task. Our findings suggest that simultaneous and sequential interactions are influenced by design-related activities including the deliberation, exchange, and integration of the group ideas, which form the main body of the collaborative task. Build on many previous studies on the collaboration over shared multi-touch devices (e.g. [19,20,36,41]), our observation further looks into when and how people interact sequentially and simultaneously, particularly from social, spatial, temporal and gestural perspectives. Building on the analysis in the previous section of simultaneous and sequential interaction within the four collaboration factors, this section puts forth observations based on the findings.

7.1Supporting Collaboration Techniques

Sequential interaction is the dominant collaboration technique for creative design with multi-user touch display. Participants spent more time on sequential interaction with the display compared to the simultaneous. The digital collaborative space is used as a supporting tool to help explain and exchange ideas. However, the conversations during the design, which mostly happens in a turn-taking fashion, influence the physical interactions.

Prior studies focus on the form of coupling styles [20,38] or technology support for both close-couple and loosely-coupled collaboration [4,19,41]. This study further explores how coupling styles affect simultaneous and sequential use of the device. When closely collaborate with each other and have tense conversation, the participants tend to use the multi-user display in a sequential mode to avoid interference. But when working in distributed attention, collaborators are more flexible suggests that though the multi-user touch-sensitive device provides the ability to handle concurrent user inputs, the interaction manner does not totally lie on the affordances in selecting interaction manner. This implies that the influence of the social and attentional factors to the physical interaction is considerable.

The participants in our study perceive the openness and freedom of the interaction space as useful to foster creativity. The equal and simultaneous access to the touch table motivates participation, thereafter encourage making contributions. Also, the participants are ready to show their findings and learn what others are doing. Though resulting in the sequential interactions with the table, the close-coupled collaboration enhances the mutual awareness to each other’s activities, and therefore builds the common ground to the collaborative task.

7.2Supporting Task Change in Interaction Stages

Prior research gave general descriptions of when the simultaneous and sequential interactions happen in different contexts [7,16,31]. Grounded in these findings, this study conducted a deeper investigation of the two interaction modes in three interaction stages. Upon our observation, we notice that more frequently the simultaneous interactions happen at the browsing stage. In the organization and specification stage when the idea is getting more and more certain, collaborators do not interact with the display at the same time. This suggests that the simultaneous browsing helps collaborators quickly acquire enough information through exploration, and generate many potentially interesting if yet unclear ideas. Sequential interaction helps collaborators pay attention to each other and maintain the same understanding towards the decided plan.

Implications from this finding include the design of the groupware on the multi-user touch display should supports simultaneous information exploration and sequential decision-making. The interactive objects should be available to all collaborators for exploratory manipulation and the spatial interferences between them should be avoided. As the design becomes clearer, mechanisms to coordinate the turn-taking idea exchange and decision-making should be considered to facilitate mutual awareness.

Our participants took the digital cards as a media to forage and make sense with the design information. Building on prior knowledge on sense-making process [29] and reflective practices with the design materials [33,39], our study suggests that different constitution and different interaction modes of the digital materials influence the path of design throughout the three design stages. When collaborating through groupware, providing abundant while simultaneously accessible materials at the beginning will expand the idea inventory. In completing the design, methods such as highlighting important objects and taking turns will encourage collaborators to focus more on the key thoughts and not diverging with too many unrelated ideas.

7.3Supporting Interactions in Territories

Prior work has explored how collaborators work in personal and shared territories, and how technology can help and hinder work activities. When working on shared tasks, participants seemed to avoid touching the display at the same time, while when working separately they interacted with digital objects simultaneously. This observation of territoriality suggests that sequential use of the objects dominate interactions within group territory and the simultaneous interaction dominates personal space.

These interaction patterns suggest that collaboration tools should support transition between personal and group work, including by reflecting recent changes to assist collaborators in observing, evaluating, and responding to modifications [3]. Technology should enable collaborators to distribute work in the space and finish simultaneously to increase efficiency of collaboration and encourage balanced participation.

Control and availability of shared objects are critical for collaborative systems [34,45]. In the context of co-located collaboration technologies, operations that interfere with personal control (e.g., mode switching) can cause interaction issues. Our observations showed that users might switch mode during personal work, ignoring global effects on others. Yet participants need availability of personal space: it facilitates idea organization, and allows the participants to work on their own subtask when others are editing the shared space. In crafting or choosing a collaborative digital table and software, one should consider ways to ensure an always-on personal space for each collaborator; or when the personal space is not available there should be proper support for awareness.

7.4Supporting Semantic Actions

Some actions are better performed simultaneously, while others may be only suitable in turns. In the observation of the semantic actions, we noticed that activities like emphasizing one card or linking a pair of cards are conducted in turns. Some actions such as sorting the cards are usually performed in parallel. Other actions include grouping and deleting can be sequential or simultaneous, based on the content and importance of the target card.

Regarding relative utility, it would be beneficial to take the interaction mode into consideration when designing the interaction technique for the multi-user touch display. For example, in the gesture based collaboration system, gestures for sorting personal objects might be preferred to be performed simultaneously, so the system should be able to handle the simultaneous gesturing during these actions. Gestures for emphasizing or connecting the digital objects should be designed in a more noticeable manner to ensure group awareness of personal thoughts [44].

The digital cards capture most affordances of the paper cards, such as moving, rotating and clustering, but the malleable and distributable feature further enhance the co-located collaboration by helping idea exchange and integration. In our observational study, the resizing and the drag-and-drop action are used to communicate interesting cards and integrate group ideas. In addition to the semantic interactions that designed for object manipulation [5,8,24,36], our study suggests that leveraging the communicational meaning of the semantic actions can also increase the mutual awareness of individual ideas and therefore reduce the effort to integrate group opinions. Other semantic actions on digital objects, such as the batch operations, should also be considered to facilitate organization and simplification of the design ideas.

This paper presents an observational study and its findings focused on collaborative interaction with the multi-user touch display. The simultaneous interaction and sequential interaction during the design activity are examined with four collaboration factors: coupling style, interaction stage, territoriality, and semantic action. The results suggest that in the design activity, sequential use of the digital objects is the major interaction manner with the multi-user touch display. Other results point to the understanding of the simultaneous and sequential patterns in different interaction stages, among different territories and upon actions with different semantic meanings. Main conclusions are as follows:

• Despite providing technology and encouragement in support of simultaneous interaction, in this study sequential interaction was the dominant collaboration technique with the multi-user touch display.

• The rich and simultaneously manipulable digital materials facilitate idea generation, but the organization and synthesization of the ideas require less interruptive and distractive approach to collaborate over the shared materials.

• The personal and collaborative spaces have their own characteristics of simultaneous or sequential access – personal space is preferred to be always and simultaneously available, while the collaborative space should be sequentially controlled.

Some actions are better performed simultaneously for efficient collaboration, while others may be only suitable to be performed in turns to better communicate ideas. Attaching the communicational meanings to the semantic gesturing enhances the mutual awareness and mutual understanding.

With the prevalence of large-scale multi-touch displays, increasingly more collaborative work has potential to be supported by technology-enhanced materials and approaches. With the lessons learned in this study, future work should address the following topics.

First, an understanding of how simultaneous or sequential interactions affect collaborative activity is not yet clear. Future study should probe the two modes of the interaction by examining how each mode increases use of the actions, reduces the interaction error, enhances the awareness of other’s activity, and so forth. The interactions probed in this study comprise basic and common touch gestures like moving, rotating and scaling the digital cards. Ways that other domain- or context-specific gesturing should be performed (and supported by digital tools) either simultaneously or sequentially needs more investigation.

Further, since sequential and simultaneous interaction plays different roles in design activities, better interaction techniques should be investigated to support the negotiations and mutual awareness among interactions with the different collaboration manners. For example, implementing semantically obvious and conflict-free gestures on the multi-touch display can help collaborators quickly understand each other’s ideas with minimal interference. A deeper examination on the communicational meaning of touch gestures and designing notification support for sharing information with gestures will contribute to better collaborative touch interactions in the design and sense-making activities.

Also, collaborative interface bearing the simultaneous use of personal space and the turn-taking access to the shared objects should be explored and assessed. Possible solutions include dividing the interaction space into private and collaborative sections and providing specific control techniques (such as special gesturing or on-screen widgets) to coordinate the access and availability of different working spaces. The design of such techniques will also reflect the nature of the collaborative tasks in different activity stages. Inspiring ideas at the beginning and focusing the group on core components towards task completion should guide the design of collaborative interfaces for multitouch tables.