Universal Document Processing

Home

Discover

Bookmarks

Keep Up With Latest Trending Papers. Computer Science, AI and Machine Learning and more.Subscribe

Added to collectionTop 100 Papers By Signal Impressions

Artificial Intelligence

Computation and Language

Universal Document Processing

Unifying Vision, Text, and Layout for Universal Document Processing

We propose universal document processing (udop), a foundation document artificial intelligence model which unifies text, image, and layout modalities together with varied task formats, including document understanding and generation.

Udop is pretrained on both large-scale unlabeled document corpora using innovative self-supervised objectives and diverse labeled data, and unifies pretraining and multi-domain downstream tasks into a prompt-based sequence generation scheme.

Our method sets the state-of-the-art on 9 document ai tasks, e.g., document understanding and qa, across diverse data domains like finance reports, academic papers, and web sites.

Authors

Zineng Tang, Ziyi Yang, Guoxin Wang, Yuwei Fang, Yang Liu, Chenguang Zhu, Michael Zeng, Cha Zhang, Mohit Bansal

Related Topics

Introduction

Document data have a 2-dimensional spatial layout : text content is structurally spread around in different locations based on diverse document types and formats (e.g., invoices vs. tax forms) ; formatted data such as figures, tables and plots are laid out across the document.

Effectively and efficiently modeling and understanding the layout is vital for document information extraction and content understanding, for example, title/signature extraction, fraudulent check detection, table processing, document classification, and automatic data entry from documents.

Document artificial intelligence studies information extraction, understanding, and analysis of digital documents, e.g., business invoices, tax forms, academic papers, etc.

It is a multimodal task where text is structurally embedded in documents, together with other vision information like symbols, figures, and style.

Document data have a strong cross-modal interactions between text and visual modalities, because the text modality is visually-situated in an image.

To address these challenges, we propose universal document processing (udop), a foundation document ai model that unifies vision, text, and layout and different document tasks.

We propose to model them with the uniform layout-induced representation (: in the input stage, we add embeddings of text tokens with the features of the image patch where the tokens are located).

To form a uniform paradigm for different vision, text, layout tasks, udop first builds a homogeneous vocabulary for texts and document layout that converts layout, i.e.

Bounding boxes, to discretized tokens.

Second, we propose vision-text-layout (vtl) transformer, consisting of a modality-agnostic encoder, text-layout decoder and vision decoder.

Result

In this work, we propose a foundation model for document artificial intelligence (document ai), which unifies the vision, text and layout modalities of documents by utilizing their strong spatial correlations through layout-induced vision-text representations and vision-text-layout transformer.

It also unites all self-supervised and supervised document tasks with a generative framework.

For the first time in document ai, we achieve customizable realistic document generation and editing.

We achieve sota on 9 tasks and currently ranks the 1st place on the document understanding benchmark leaderboard.

Read the Paper

Read Paper

Bookmarks

Bookmarks

Keep Up With Latest Trending Papers. Computer Science, AI and Machine Learning and more.Subscribe

Added to collectionTop 100 Papers By Signal Impressions

Artificial Intelligence

Computation and Language

Universal Document Processing

Unifying Vision, Text, and Layout for Universal Document Processing

Our method sets the state-of-the-art on 9 document ai tasks, e.g., document understanding and qa, across diverse data domains like finance reports, academic papers, and web sites.

Authors

Zineng Tang, Ziyi Yang, Guoxin Wang, Yuwei Fang, Yang Liu, Chenguang Zhu, Michael Zeng, Cha Zhang, Mohit Bansal

Related Topics

Introduction

Document artificial intelligence studies information extraction, understanding, and analysis of digital documents, e.g., business invoices, tax forms, academic papers, etc.

It is a multimodal task where text is structurally embedded in documents, together with other vision information like symbols, figures, and style.

Document data have a strong cross-modal interactions between text and visual modalities, because the text modality is visually-situated in an image.

To address these challenges, we propose universal document processing (udop), a foundation document ai model that unifies vision, text, and layout and different document tasks.

We propose to model them with the uniform layout-induced representation (: in the input stage, we add embeddings of text tokens with the features of the image patch where the tokens are located).

To form a uniform paradigm for different vision, text, layout tasks, udop first builds a homogeneous vocabulary for texts and document layout that converts layout, i.e.

Bounding boxes, to discretized tokens.

Second, we propose vision-text-layout (vtl) transformer, consisting of a modality-agnostic encoder, text-layout decoder and vision decoder.

Result

It also unites all self-supervised and supervised document tasks with a generative framework.

For the first time in document ai, we achieve customizable realistic document generation and editing.

We achieve sota on 9 tasks and currently ranks the 1st place on the document understanding benchmark leaderboard.

Read the Paper

Read Paper

Home

Discover

Bookmarks

◐ Latest Activity

◐ Recommended Follows