README.md
3.7 KB · 87 lines · markdown Raw
1 ---
2 license: apache-2.0
3 language: en
4 datasets:
5 - pubmed
6 tags:
7 - bert
8 - exbert
9 - linkbert
10 - biolinkbert
11 - feature-extraction
12 - fill-mask
13 - question-answering
14 - text-classification
15 - token-classification
16 widget:
17 - text: "Sunitinib is a tyrosine kinase inhibitor"
18 ---
19
20 ## BioLinkBERT-large
21
22 BioLinkBERT-large model pretrained on [PubMed](https://pubmed.ncbi.nlm.nih.gov/) abstracts along with citation link information. It is introduced in the paper [LinkBERT: Pretraining Language Models with Document Links (ACL 2022)](https://arxiv.org/abs/2203.15827). The code and data are available in [this repository](https://github.com/michiyasunaga/LinkBERT).
23
24 This model achieves state-of-the-art performance on several biomedical NLP benchmarks such as [BLURB](https://microsoft.github.io/BLURB/) and [MedQA-USMLE](https://github.com/jind11/MedQA).
25
26
27 ## Model description
28
29 LinkBERT is a transformer encoder (BERT-like) model pretrained on a large corpus of documents. It is an improvement of BERT that newly captures **document links** such as hyperlinks and citation links to include knowledge that spans across multiple documents. Specifically, it was pretrained by feeding linked documents into the same language model context, besides a single document.
30
31 LinkBERT can be used as a drop-in replacement for BERT. It achieves better performance for general language understanding tasks (e.g. text classification), and is also particularly effective for **knowledge-intensive** tasks (e.g. question answering) and **cross-document** tasks (e.g. reading comprehension, document retrieval).
32
33
34 ## Intended uses & limitations
35
36 The model can be used by fine-tuning on a downstream task, such as question answering, sequence classification, and token classification.
37 You can also use the raw model for feature extraction (i.e. obtaining embeddings for input text).
38
39
40 ### How to use
41
42 To use the model to get the features of a given text in PyTorch:
43
44 ```python
45 from transformers import AutoTokenizer, AutoModel
46 tokenizer = AutoTokenizer.from_pretrained('michiyasunaga/BioLinkBERT-large')
47 model = AutoModel.from_pretrained('michiyasunaga/BioLinkBERT-large')
48 inputs = tokenizer("Sunitinib is a tyrosine kinase inhibitor", return_tensors="pt")
49 outputs = model(**inputs)
50 last_hidden_states = outputs.last_hidden_state
51 ```
52
53 For fine-tuning, you can use [this repository](https://github.com/michiyasunaga/LinkBERT) or follow any other BERT fine-tuning codebases.
54
55
56 ## Evaluation results
57
58 When fine-tuned on downstream tasks, LinkBERT achieves the following results.
59
60 **Biomedical benchmarks ([BLURB](https://microsoft.github.io/BLURB/), [MedQA](https://github.com/jind11/MedQA), [MMLU](https://github.com/hendrycks/test), etc.):** BioLinkBERT attains new state-of-the-art.
61
62 | | BLURB score | PubMedQA | BioASQ | MedQA-USMLE |
63 | ---------------------- | -------- | -------- | ------- | -------- |
64 | PubmedBERT-base | 81.10 | 55.8 | 87.5 | 38.1 |
65 | **BioLinkBERT-base** | **83.39** | **70.2** | **91.4** | **40.0** |
66 | **BioLinkBERT-large** | **84.30** | **72.2** | **94.8** | **44.6** |
67
68 | | MMLU-professional medicine |
69 | ---------------------- | -------- |
70 | GPT-3 (175 params) | 38.7 |
71 | UnifiedQA (11B params) | 43.2 |
72 | **BioLinkBERT-large (340M params)** | **50.7** |
73
74
75 ## Citation
76
77 If you find LinkBERT useful in your project, please cite the following:
78
79 ```bibtex
80 @InProceedings{yasunaga2022linkbert,
81 author = {Michihiro Yasunaga and Jure Leskovec and Percy Liang},
82 title = {LinkBERT: Pretraining Language Models with Document Links},
83 year = {2022},
84 booktitle = {Association for Computational Linguistics (ACL)},
85 }
86 ```
87