README.md · OpenELM-1_1B-Instruct

1

---

2

license: apple-amlr

3

license_name: apple-sample-code-license

4

license_link: LICENSE

5

---

6

7

# OpenELM

8

9

*Sachin Mehta, Mohammad Hossein Sekhavat, Qingqing Cao, Maxwell Horton, Yanzi Jin, Chenfan Sun, Iman Mirzadeh, Mahyar Najibi, Dmitry Belenko, Peter Zatloukal, Mohammad Rastegari*

10

11

We introduce **OpenELM**, a family of **Open** **E**fficient **L**anguage **M**odels. OpenELM uses a layer-wise scaling strategy to efficiently allocate parameters within each layer of the transformer model, leading to enhanced accuracy. We pretrained OpenELM models using the [CoreNet](https://github.com/apple/corenet) library. We release both pretrained and instruction tuned models with 270M, 450M, 1.1B and 3B parameters. We release the complete framework, encompassing data preparation, training, fine-tuning, and evaluation procedures, alongside multiple pre-trained checkpoints and training logs, to facilitate open research.

12

13

Our pre-training dataset contains RefinedWeb, deduplicated PILE, a subset of RedPajama, and a subset of Dolma v1.6, totaling approximately 1.8 trillion tokens. Please check license agreements and terms of these datasets before using them.

14

15

16

17

## Usage

18

19

We have provided an example function to generate output from OpenELM models loaded via [HuggingFace Hub](https://huggingface.co/docs/hub/) in `generate_openelm.py`.

20

21

You can try the model by running the following command:

22

```

23

python generate_openelm.py --model apple/OpenELM-1_1B-Instruct --hf_access_token [HF_ACCESS_TOKEN] --prompt 'Once upon a time there was' --generate_kwargs repetition_penalty=1.2

24

```

25

Please refer to [this link](https://huggingface.co/docs/hub/security-tokens) to obtain your hugging face access token.

26

27

Additional arguments to the hugging face generate function can be passed via `generate_kwargs`. As an example, to speedup the inference, you can try [lookup token speculative generation](https://huggingface.co/docs/transformers/generation_strategies) by passing the `prompt_lookup_num_tokens` argument as follows:

28

```

29

python generate_openelm.py --model apple/OpenELM-1_1B-Instruct --hf_access_token [HF_ACCESS_TOKEN] --prompt 'Once upon a time there was' --generate_kwargs repetition_penalty=1.2 prompt_lookup_num_tokens=10

30

```

31

Alternatively, try model-wise speculative generation with an [assistive model](https://huggingface.co/blog/assisted-generation) by passing a smaller model through the `assistant_model` argument, for example:

32

```

33

python generate_openelm.py --model apple/OpenELM-1_1B-Instruct --hf_access_token [HF_ACCESS_TOKEN] --prompt 'Once upon a time there was' --generate_kwargs repetition_penalty=1.2 --assistant_model [SMALLER_MODEL]

34

```

35

36

## Main Results

37

38

### Zero-Shot

39

40

| **Model Size**                                                              | **ARC-c** | **ARC-e** | **BoolQ** | **HellaSwag** | **PIQA**  | **SciQ**  | **WinoGrande** | **Average** |

41

|-----------------------------------------------------------------------------|-----------|-----------|-----------|---------------|-----------|-----------|----------------|-------------|

42

| [OpenELM-270M](https://huggingface.co/apple/OpenELM-270M)                   | 26.45     | 45.08     | **53.98** | 46.71         | 69.75     | **84.70** | **53.91**      | 54.37       |

43

| [OpenELM-270M-Instruct](https://huggingface.co/apple/OpenELM-270M-Instruct) | **30.55** | **46.68** | 48.56     | **52.07**     | **70.78** | 84.40     | 52.72          | **55.11**   |

44

| [OpenELM-450M](https://huggingface.co/apple/OpenELM-450M)                   | 27.56     | 48.06     | 55.78     | 53.97         | 72.31     | 87.20     | 58.01          | 57.56       |

45

| [OpenELM-450M-Instruct](https://huggingface.co/apple/OpenELM-450M-Instruct) | **30.38** | **50.00** | **60.37** | **59.34**     | **72.63** | **88.00** | **58.96**      | **59.95**   |

46

| [OpenELM-1_1B](https://huggingface.co/apple/OpenELM-1_1B)                   | 32.34     | **55.43** | 63.58     | 64.81         | **75.57** | **90.60** | 61.72          | 63.44       |

47

| [OpenELM-1_1B-Instruct](https://huggingface.co/apple/OpenELM-1_1B-Instruct) | **37.97** | 52.23     | **70.00** | **71.20**     | 75.03     | 89.30     | **62.75**      | **65.50**   |

48

| [OpenELM-3B](https://huggingface.co/apple/OpenELM-3B)                       | 35.58     | 59.89     | 67.40     | 72.44         | 78.24     | **92.70** | 65.51          | 67.39       |

49

| [OpenELM-3B-Instruct](https://huggingface.co/apple/OpenELM-3B-Instruct)     | **39.42** | **61.74** | **68.17** | **76.36**     | **79.00** | 92.50     | **66.85**      | **69.15**   |

50

51

### LLM360

52

53

| **Model Size**                                                              | **ARC-c** | **HellaSwag** | **MMLU**  | **TruthfulQA** | **WinoGrande** | **Average** |

54

|-----------------------------------------------------------------------------|-----------|---------------|-----------|----------------|----------------|-------------|

55

| [OpenELM-270M](https://huggingface.co/apple/OpenELM-270M)                   | 27.65     | 47.15         | 25.72     | **39.24**      | **53.83**      | 38.72       |

56

| [OpenELM-270M-Instruct](https://huggingface.co/apple/OpenELM-270M-Instruct) | **32.51** | **51.58**     | **26.70** | 38.72          | 53.20          | **40.54**   |

57

| [OpenELM-450M](https://huggingface.co/apple/OpenELM-450M)                   | 30.20     | 53.86         | **26.01** | 40.18          | 57.22          | 41.50       |

58

| [OpenELM-450M-Instruct](https://huggingface.co/apple/OpenELM-450M-Instruct) | **33.53** | **59.31**     | 25.41     | **40.48**      | **58.33**      | **43.41**   |

59

| [OpenELM-1_1B](https://huggingface.co/apple/OpenELM-1_1B)                   | 36.69     | 65.71         | **27.05** | 36.98          | 63.22          | 45.93       |

60

| [OpenELM-1_1B-Instruct](https://huggingface.co/apple/OpenELM-1_1B-Instruct) | **41.55** | **71.83**     | 25.65     | **45.95**      | **64.72**      | **49.94**   |

61

| [OpenELM-3B](https://huggingface.co/apple/OpenELM-3B)                       | 42.24     | 73.28         | **26.76** | 34.98          | 67.25          | 48.90       |

62

| [OpenELM-3B-Instruct](https://huggingface.co/apple/OpenELM-3B-Instruct)     | **47.70** | **76.87**     | 24.80     | **38.76**      | **67.96**      | **51.22**   |

63

64

65

### OpenLLM Leaderboard

66

67

| **Model Size**                                                              | **ARC-c** | **CrowS-Pairs** | **HellaSwag** | **MMLU**  | **PIQA**  | **RACE**  | **TruthfulQA** | **WinoGrande** | **Average** |

68

|-----------------------------------------------------------------------------|-----------|-----------------|---------------|-----------|-----------|-----------|----------------|----------------|-------------|

69

| [OpenELM-270M](https://huggingface.co/apple/OpenELM-270M)                   | 27.65     | **66.79**       | 47.15         | 25.72     | 69.75     | 30.91     | **39.24**      | **53.83**      | 45.13       |

70

| [OpenELM-270M-Instruct](https://huggingface.co/apple/OpenELM-270M-Instruct) | **32.51** | 66.01           | **51.58**     | **26.70** | **70.78** | 33.78     | 38.72          | 53.20          | **46.66**   |

71

| [OpenELM-450M](https://huggingface.co/apple/OpenELM-450M)                   | 30.20     | **68.63**       | 53.86         | **26.01** | 72.31     | 33.11     | 40.18          | 57.22          | 47.69       |

72

| [OpenELM-450M-Instruct](https://huggingface.co/apple/OpenELM-450M-Instruct) | **33.53** | 67.44           | **59.31**     | 25.41     | **72.63** | **36.84** | **40.48**      | **58.33**      | **49.25**   |

73

| [OpenELM-1_1B](https://huggingface.co/apple/OpenELM-1_1B)                   | 36.69     | **71.74**       | 65.71         | **27.05** | **75.57** | 36.46     | 36.98          | 63.22          | 51.68       |

74

| [OpenELM-1_1B-Instruct](https://huggingface.co/apple/OpenELM-1_1B-Instruct) | **41.55** | 71.02           | **71.83**     | 25.65     | 75.03     | **39.43** | **45.95**      | **64.72**      | **54.40**   |

75

| [OpenELM-3B](https://huggingface.co/apple/OpenELM-3B)                       | 42.24     | **73.29**       | 73.28         | **26.76** | 78.24     | **38.76** | 34.98          | 67.25          | 54.35       |

76

| [OpenELM-3B-Instruct](https://huggingface.co/apple/OpenELM-3B-Instruct)     | **47.70** | 72.33           | **76.87**     | 24.80     | **79.00** | 38.47     | **38.76**      | **67.96**      | **55.73**   |

77

78

See the technical report for more results and comparison.

79

80

## Evaluation

81

82

### Setup

83

84

Install the following dependencies:

85

86

```bash

87

88

# install public lm-eval-harness

89

90

harness_repo="public-lm-eval-harness"

91

git clone https://github.com/EleutherAI/lm-evaluation-harness ${harness_repo}

92

cd ${harness_repo}

93

# use main branch on 03-15-2024, SHA is dc90fec

94

git checkout dc90fec

95

pip install -e .

96

cd ..

97

98

# 66d6242 is the main branch on 2024-04-01

99

pip install datasets@git+https://github.com/huggingface/datasets.git@66d6242

100

pip install tokenizers>=0.15.2 transformers>=4.38.2 sentencepiece>=0.2.0

101

102

```

103

104

### Evaluate OpenELM

105

106

```bash

107

108

# OpenELM-1_1B-Instruct

109

hf_model=apple/OpenELM-1_1B-Instruct

110

111

# this flag is needed because lm-eval-harness set add_bos_token to False by default, but OpenELM uses LLaMA tokenizer which requires add_bos_token to be True

112

tokenizer=meta-llama/Llama-2-7b-hf

113

add_bos_token=True

114

batch_size=1

115

116

mkdir lm_eval_output

117

118

shot=0

119

task=arc_challenge,arc_easy,boolq,hellaswag,piqa,race,winogrande,sciq,truthfulqa_mc2

120

lm_eval --model hf \

121

        --model_args pretrained=${hf_model},trust_remote_code=True,add_bos_token=${add_bos_token},tokenizer=${tokenizer} \

122

--tasks ${task} \

123

--device cuda:0 \

124

--num_fewshot ${shot} \

125

--output_path ./lm_eval_output/${hf_model//\//_}_${task//,/_}-${shot}shot \

126

--batch_size ${batch_size} 2>&1 | tee ./lm_eval_output/eval-${hf_model//\//_}_${task//,/_}-${shot}shot.log

127

128

shot=5

129

task=mmlu,winogrande

130

lm_eval --model hf \

131

        --model_args pretrained=${hf_model},trust_remote_code=True,add_bos_token=${add_bos_token},tokenizer=${tokenizer} \

132

--tasks ${task} \

133

--device cuda:0 \

134

--num_fewshot ${shot} \

135

--output_path ./lm_eval_output/${hf_model//\//_}_${task//,/_}-${shot}shot \

136

--batch_size ${batch_size} 2>&1 | tee ./lm_eval_output/eval-${hf_model//\//_}_${task//,/_}-${shot}shot.log

137

138

shot=25

139

task=arc_challenge,crows_pairs_english

140

lm_eval --model hf \

141

        --model_args pretrained=${hf_model},trust_remote_code=True,add_bos_token=${add_bos_token},tokenizer=${tokenizer} \

142

--tasks ${task} \

143

--device cuda:0 \

144

--num_fewshot ${shot} \

145

--output_path ./lm_eval_output/${hf_model//\//_}_${task//,/_}-${shot}shot \

146

--batch_size ${batch_size} 2>&1 | tee ./lm_eval_output/eval-${hf_model//\//_}_${task//,/_}-${shot}shot.log

147

148

shot=10

149

task=hellaswag

150

lm_eval --model hf \

151

        --model_args pretrained=${hf_model},trust_remote_code=True,add_bos_token=${add_bos_token},tokenizer=${tokenizer} \

152

--tasks ${task} \

153

--device cuda:0 \

154

--num_fewshot ${shot} \

155

--output_path ./lm_eval_output/${hf_model//\//_}_${task//,/_}-${shot}shot \

156

--batch_size ${batch_size} 2>&1 | tee ./lm_eval_output/eval-${hf_model//\//_}_${task//,/_}-${shot}shot.log

157

158

```

159

160

161

## Bias, Risks, and Limitations

162

163

The release of OpenELM models aims to empower and enrich the open research community by providing access to state-of-the-art language models. Trained on publicly available datasets, these models are made available without any safety guarantees. Consequently, there exists the possibility of these models producing outputs that are inaccurate, harmful, biased, or objectionable in response to user prompts. Thus, it is imperative for users and developers to undertake thorough safety testing and implement appropriate filtering mechanisms tailored to their specific requirements.

164

165

## Citation

166

167

If you find our work useful, please cite:

168

169

```BibTex

170

@article{mehtaOpenELMEfficientLanguage2024,

171

title = {{OpenELM}: {An} {Efficient} {Language} {Model} {Family} with {Open} {Training} and {Inference} {Framework}},

172

shorttitle = {{OpenELM}},

173

url = {https://arxiv.org/abs/2404.14619v1},

174

language = {en},

175

urldate = {2024-04-24},

176

journal = {arXiv.org},

177

	author = {Mehta, Sachin and Sekhavat, Mohammad Hossein and Cao, Qingqing and Horton, Maxwell and Jin, Yanzi and Sun, Chenfan and Mirzadeh, Iman and Najibi, Mahyar and Belenko, Dmitry and Zatloukal, Peter and Rastegari, Mohammad},

178

month = apr,

179

year = {2024},

180

}

181

182

@inproceedings{mehta2022cvnets,

183

author = {Mehta, Sachin and Abdolhosseini, Farzad and Rastegari, Mohammad},

184

title = {CVNets: High Performance Library for Computer Vision},

185

year = {2022},

186

booktitle = {Proceedings of the 30th ACM International Conference on Multimedia},

187

series = {MM '22}

188

}

189

```

190