We present a new solution towards building a crowd knowledge enhanced multimodal conversational system for travel. It aims to assist users in completing various travel-related tasks, such as searching for restaurants or things to do, in a multimodal conversation manner involving both text and images. In order to achieve this goal, we ground this research on the combination of multimodal understanding and recommendation techniques which explores the possibility of a more convenient information seeking paradigm. Specifically, we build the system in a modular manner where each modular construction is enriched with crowd knowledge from social sites. To the best of our knowledge, this is the first work that attempts to build intelligent multimodal conversational systems for travel, and moves an important step towards developing human-like assistants for completion of daily life tasks. Several current challenges are also pointed out as our future directions.

Authors and Affiliations

1. NGS, National University of Singapore, Singapore, Singapore

   Lizi Liao & Tat-Seng Chua

2. FXPAL, Palo Alto, USA

   Lyndon Kennedy & Lynn Wilcox

Corresponding author

Correspondence toLizi Liao.

Editors and Affiliations

1. Korea Advanced Institute of Science and, Daejeon, Korea (Republic of)

   Yong Man Ro

2. National Chiao Tung University, Hsinchu, Taiwan

   Wen-Huang Cheng

3. Korea Advanced Institute of Science and Technology, Daejeon, Korea (Republic of)

   Junmo Kim

4. National Cheng Kung University, Tainan City, Taiwan

   Wei-Ta Chu

5. Tsinghua University, Beijing, China

   Peng Cui

6. Korea Advanced Institute of Science and Technology, Daejeon, Korea (Republic of)

   Jung-Woo Choi

7. National Tsing Hua University, Hsinchu, Taiwan

   Min-Chun Hu

8. Ghent University, Ghent, Belgium

   Wesley De Neve

Appendix A: State Tracking

State tracking refers to the maintenance of the dialogue state\(\mathcal {S}_t\) which is the representation of the conversation session until timet. Based on the state\(\mathcal {S}_{t-1}\) in former time step and the multimodal understanding result\(\mathcal {U}_t\) for utterance in time stept, the dialogue state is obtained as follows:

where\(\mathcal {G}\) refers to a set of rules. We generally summarize the rules as below:

1. (1)

   if\(\mathcal {M}_t = Chitchat\), then\(\mathcal {S}_t = \mathcal {S}_{t-1}\);

2. (2)

   if domain\(\mathcal {D}_t\) is changed:\(\mathcal {S}_t\) will be updated totally based on\(\mathcal {U}_t\);

3. (3)

   if domain\(\mathcal {D}_t\) is not changed: if\(\mathcal {M}_t \ne Negation\),\(\mathcal {S}_t\) will inherit information stored in\(\mathcal {S}_{t-1}\);

4. (4)

   if domain\(\mathcal {D}_t\) is not changed: if\(\mathcal {M}_t = Negation\),\(\mathcal {S}_t\) will inherit information stored in\(\mathcal {S}_{t-1}\) while update the parts according\(\mathcal {U}_t\);

5. (5)

   if the time interval between two consecutive utterances exceeds a pre-defined length at timet, then\(\mathcal {S}_t\) will be cleaned.

Tracking dialogue states is the key to elevate user experience on multi-turn conversation. The main reason that we do not follow the previous works to learn models is because of the lack of dialogue data to train statistical tracking models. We leave leveraging session-level labeled dialogue data to improve the state tracking task as our future work.

Appendix B: Action Decision

At each turn of conversation between user and agent, the dialogue management module takes the current state tracking results as input, and outputs the corresponding actions. Due to the lack of large-scale dialogue training data, we also resort to a set of rules. Intuitively, the main action types considered and the conditions for triggering it are as below:

• Proactive Questioning. This action will be triggered when (a) a recommendation intent is detected, (b) a domain is detected, and (c) no enough constraints or attributes is detected in\(\mathcal {S}_t\). This action is often used to obtain more constraints or attributes to narrow down the search space.

• Candidate Listing. This action is often triggered when recommendation results are obtained or theShow more intent is detected. As each venue in our dataset is associated with Foursquare photos, we implement candidate listing via a list of images where each image corresponds to a venue. In the interface, user can conveniently choose a venue by simply clicking its corresponding image.

• Venue Recommendation. This action is triggered when the intent\(\mathcal {I}_t\) in\(\mathcal {S}_t\) isRecommendation, and it will retrieve results from the recommendation module.

• Question Answering. It will be triggered when a venue is selected and one of its slot names are detected in\(\mathcal {U}_t\) without value around. It returns the missing attribute value by looking up the venue database.

• Review Summary. This action will be triggered when a venue is selected and the intent\(\mathcal {I}_t\) in\(\mathcal {S}_t\) isAsk opinion. It will summarize the reviews of the target venue and present in organized form.

• API call. It will be triggered when a venue is selected and theMap direction intent is present in\(\mathcal {I}_t\). The Google Map API will be called with the start position and destination. Currently, only the Map API is integrated. However, other APIs such as weather report can also be integrated with proper modifications.

• Chitchat. This action will be triggered when none travel venue seeking related intent is detected. As pointed out by [18], nearly 80% of utterances are chitchat queries for e-commerce bots. If the system cannot reply to them, then the conversation may not be able to continue. Thus, it will activate the chitchat response generation to obtain a reply.

Reprints and permissions

© 2020 Springer Nature Switzerland AG

Policies and ethics