MultiInstruct: A Benchmark for Multimodal Instruction Tuning

Home

Discover

Bookmarks

Keep Up With Latest Trending Papers. Computer Science, AI and Machine Learning and more.Subscribe

Authors

Zhiyang Xu, Ying Shen, Lifu Huang

Related Topics

Introduction

Instruction tuning has achieved significant success in zero-shot learning on natural language processing tasks.

By fine-tuning a pre-trained multimodal language model on tasks described through instructions, instruction tuning allows the model to learn to understand and follow the instructions to perform predictions on unseen tasks.

In this work, we propose, the first benchmark dataset for multimodal instruction tuning with 47 diverse tasks from 11 broad categories that require visual understanding and multimodal reasoning.

We formulate all the tasks into a unified sequence-to-sequence format in which the input text, images, instructions, and bounding boxes are represented in the same token space.

For each task, we create at least 5,000 instances and 5 instructions that are manually written by two experts in natural language processing.

Experimental results demonstrate strong zero-shot performance on various unseen multimodal tasks with instruction tuning and the potential of further improving it by leveraging large-scale text-only instruction datasets.

Result

Multimodal instruction tuning is a key question in instruction guidance and zero-shot performance on multimodal tasks.

One key question is how to effectively leverage the large-scale text-only instruction datasets to enhance the instruction guidance, zero-short performance, and performance on all the unseen evaluation tasks and metrics.

Here we show that simply fine-tuning the original pre-trained multimodal language model (ofa) on actually degrades the model s zero-shooting performance, as shown by comparing the model only trained on the mixture of and, which is possibly due to the imbalance of the training tasks from these two datasets.

On the other hand, simply training the model on a mixture of and does not lead to the same level of performance improvement compared with the original pre-trained ofa only trained on, demonstrating the potential benefit of the much larger text-only training datasets to multimodal instruction tuning.

Read the Paper

Read Paper

Bookmarks

Bookmarks

Keep Up With Latest Trending Papers. Computer Science, AI and Machine Learning and more.Subscribe

Authors

Zhiyang Xu, Ying Shen, Lifu Huang

Related Topics

Introduction

Instruction tuning has achieved significant success in zero-shot learning on natural language processing tasks.

In this work, we propose, the first benchmark dataset for multimodal instruction tuning with 47 diverse tasks from 11 broad categories that require visual understanding and multimodal reasoning.

We formulate all the tasks into a unified sequence-to-sequence format in which the input text, images, instructions, and bounding boxes are represented in the same token space.

For each task, we create at least 5,000 instances and 5 instructions that are manually written by two experts in natural language processing.

Result

Multimodal instruction tuning is a key question in instruction guidance and zero-shot performance on multimodal tasks.

Read the Paper

Read Paper

Home

Discover

Bookmarks

◐ Recommended Follows

◐ Latest Activity