Frequency Analysis

I. Introduction

This page describes the frequency and valid word (single words and multiwords). Frequency (word count) in the MEDLINE n-gram set are used for frequency analysis. Data of 2015 are used in the example below.

II. Word Count Class vs. Tern Number

  • Approach:
    • Split Lexicon to single words (464,781) and multiwords (431,432)
    • LMW candidates from Acronym Expansion Pattern matcher to MEDLINE n-gram set
    • Word court class: WC 100 incremental

  • Results (word-court-class vs. term number):

    • Most valid words are located in the low WC range
    • Same result as "Alice in Wonderland"

III. Word Count Class vs. Precision, Recall, F1

  • Approach:
    • Only use on LMW candidates from Acronym Expansion Pattern matcher to MEDLINE n-gram set
    • Word court class: WC 100 incremental
    • Local precision = (valid tags/total tags)

      only focus on local word count class

    • Local recall = (valid tags/total valid tags)

      Normalized to 0 ~ 1, use the max. recall as 1.

  • Results:

    • Low Frequency has higher recall and F1 score, with precision above 0.8.
    • LMW aquisition is set on the low WC range (100 - 10000)
    • LSW (Single word) is set on high WC range