The MEDLINE N-gram Set 2016: by Split, Group, Filter, and Combine Algorithm
The MEDLINE n-gram set (generated by split, group, filter, and combine algorithm) is listed as bellows. For each MEDLINE record, title and abstract are used as the source of n-grams. They are combined, tokenized into sentences, and then tokenized into tokens (words use space as word boundary). Finally, n-grams are generated by filtering out terms with more than 50 characters or the total word count is less than 30. The specifications of generating these n-grams are listed as follows:
- MEDLINE: 2016 - TI and AB (from MEDLINE Baseline Repository - MBR, PmidTiAbS16nXXXX.txt: 1 ~ 812)
- Method: Split, Combine, Filter Algorithm
- Max. Character Size: 50
- Min. word count: 30
- Min. document count: 1
- Total document count: 24,358,442
- Total sentence count: 143,471,776
- Total token count: 2,971,013,236
- N-gram files
- File format - 3 fields:
Document count Word count N-gram - Sorted by document count, word count, then alphabetic order of n-grams. N-gram set is not sorted. It can be sorted by nGramUtil package.
Download:
N-grams File Zip Size Actual Size No. of n-grams Unigrams 1-gram.2016.tgz 5.9 MB 15 MB 883,287 Bigrams 2-gram.2016.tgz 37 MB 109 MB 5,114,547 Trigrams 3-gram.2016.tgz 55 MB 182 MB 7,134,807 Four-grams 4-gram.2016.tgz 38 MB 132 MB 4,380,474 Five-grams 5-gram.2016.tgz 18 MB 64 MB 1,812,223 N-gram Set nGramSet.2016.30.tgz 153 MB 499 MB 19,325,338 Distilled N-gram Set distilledNGram.2016.tgz 60 MB 191 7,402,848 - File format - 3 fields: