Clinical note summarizers have great potential for enhancing medical documentation efficiency and have great market demand. These tools aim to alleviate the administrative burden on healthcare professionals, allowing them to focus more on patient care rather than tedious paperwork.

In an era where healthcare systems are increasingly adopting digital solutions, condensing complex and often lengthy clinical notes into concise, accurate summaries is not just an innovation but a necessity. However, creating such tools comes with significant challenges, such as hallucinations and poor-quality summaries. This blog will explore the techniques and challenges, as well as introduce our original and effective way in generating clinical note summarizations, shedding light on how our method can mitigate hallucinations in this area.

Two of the major challenges that we face when summarizing clinical notes are factual inconsistencies with the original clinical notes and omission of critical information due to the inherent nature of summarization tasks.

Factual inconsistency refers to cases where asserted facts in the summaries are actually not backed by the original clinical note. This can happen in two ways, either by presenting a completely wrong fact inconsistent with the facts in the world, or by talking about things not mentioned in the original text.

Another critical problem identified with medical data summarization is the omission of important facts, or “content hallucination”. Summarization, by its own nature, involves condensing information, and in doing so, the large language models might leave out essential details. The problem arises when these omissions are not just minor details but critical information that could lead to significant misinterpretation. As the definition of hallucination differs among authors, this issue of “content hallucination” is sometimes overlooked in works on abstractive summarization. In the context of medical applications, these errors are particularly concerning and something we should absolutely want to avoid.

These challenges lie in the fact that large language models lack domain-specific considerations to decide which fact or key words should be included in the medical summary. For illustration, consider the following sample source text and summary:

Source Text

Reference Summary

The reference summary avoids mentioning myeloma, a very serious disease, focusing instead on the immediate issues for which the patient visited. However, large pre-trained models tend to be verbose and end up mentioning the history of myeloma. This demonstrates the challenge of deciding which pieces of information to include in the summary, even when some details that should be omitted are seemingly important.

With the two problems of summary generation in mind, we have identified these three qualities that would be very desirable to have in the generated summaries.

1. Selective Inclusion and Exclusion

Implement a mechanism to control which information from the original text is included or excluded, rather than relying solely on the language model.

2. Conciseness

Reduce the length of the clinical notes while retaining essential information. This ensures that the summary is both efficient to read and effective in conveying the necessary details.

3. Preservation of Original Terminology

Maintain the original terminology used by clinicians to ensure accuracy and consistency. Since these summaries are intended for healthcare professionals who may need to refer back to them, preserving the precise language from the clinical notes is essential. Simplifying or altering terms could cause confusion and reduce their reliability as a reference.

Figure 1: Workflow of Our Pipeline

To address the challenge of generating high-quality clinical note summaries, we propose a structured pipeline comprising three essential modules: a Named Entity Recognition (NER) model, a Keyword Importance Classification model, and a large language model for generating the summaries.

We leveraged BERT-based models for the NER and keyword classification tasks and used Llama3 as the language model for inference.

The process begins with the clinical note, which serves as the input. The first step is passing this text through the NER model, which identifies and extracts relevant medical keywords, such as diagnoses, symptoms and so on.

Next, these extracted keywords, along with the original input text, are fed into the Keyword Importance Classification model. This module determines whether each keyword should be included in the final summary or not. Our next section will focus on our methodology of designing and training this module.

Finally, the selected keywords are used to prompt the large language model, instructing it to generate a summary with particular emphasis on the selected keywords. The result is a clear and focused summary tailored to the medical context and aligned with the needs of healthcare professionals.

Let’s explore the advantages of our proposed keyword selection and prompting method, and why it is particularly well-suited for summarizing clinical notes:

The key feature in our pipeline is our exploration on how to provide a high-quality list of keywords as a prompt to the language model.

1. Keywords provide critical clues for identifying the valuable content that should be included in the summary.

2. Get model to focus on the key content on top of the “summarize this” instruction.

3. Proven enhancement in summary generation.^1,2,3

Here is an example of the entities recognized by the NER model, where the keywords are highlighted:

Figure 2: Highlighted Keywords Recognized by NER Model

We need to address the question of which keywords should go into the summary. To answer this, we decided to train a keyword classification model which performs binary classification for each named entities. The class (should be included or shouldn’t be included) depends on several factors:

We built our keywords selection module on Longformer. To accommodate for the often lengthy nature of clinical notes, we decide to leverage Longformer for longer contexts. Longformer is a variant of BERT, designed with a more efficient self-attention mechanism, which allows the model to process sequences up to 4,096 tokens, ensuring that most clinical notes can be fit entirely in one window. By maintaining access to the full text context, the model can better understand the relationships between keywords and their surrounding information, leading to more accurate classification and summarization.

Meanwhile, Longformer can capture interdependencies among text, entity types, and keyword classes, as mentioned in the core problem section. By recognizing these interdependencies, the model ensures that the selected keywords are both relevant and meaningful within the clinical context.

To optimize the task for BERT’s capabilities, we structure the keyword classification task to align with BERT’s strengths, transforming it into a masked token prediction problem. This setup not only simplifies the classification process but also leverages BERT’s pretraining, which is designed to predict masked words within a sequence.

Together, these advantages ensure that our keyword selection model is  effective and practical, and that it will provide a list of high-quality keywords for the next step.

The most important, non-trivial component of this model is the entity classifier. We outline the model description and training pipeline below.

Entity Extraction

Using a dataset of doctor-written patient summaries, which includes pairs of clinical notes and the corresponding doctor-written patient summaries, and a pre-trained clinical NER model, we first extract the named entities with its classes in both the clinical note and the summary. By identifying which keywords appear in the summary, we can put a binary label on each named entity in the source text. Here, we introduce two new special tokens.

The first two mark the boundaries of a named entity within a text, so that we can clearly define where each keyword begins and ends, making it easier for the model to focus on these entities during classification.

Labeling

Meanwhile, we introduce two more new tokens to indicate if the entity should be included in the summary.

Specifically, <0> indicates that the keyword should not be included, and <1> indicates it should be included. However, the key problem in our data preparation is how to generate high-quality <0> or <1> labels, since simply labeling by exact character matching will return terrible results.

To generate high-quality labels, we’ve considered the following scenarios where we should label the entity as <1>:

1. Exact Character Matching: “dizziness” – “dizziness”
2. Acronyms: “CAD” – “Coronary Artery Disease”; “MRI” – “Magnetic Resonance Imaging”
3. Abbreviations: “trops” – “troponins”; “Dx“ – “Diagnosis“
4. Generalization: “pMIBI stress” – “a stress test”; “HCTZ” – “diuretic medication”
5. Language difference: “Heart” – “καρδιά”

These keywords that are mentioned indirectly should also count as <1>. We used the GPT-4o assistant API processing 1k clinical note/summary pairs to help identify if the keywords do appear in summary, either directly or indirectly.

Dataset Augmentation and Training Target

We augmented the clinical notes to make the format more suitable for the Longformer classification task and ensured that named entities were clearly labeled and aligned with the model’s input requirements. Here is an example:

Figure 3: Example of Augmented Clinical Note

The training process would involve masking only the special <0>or<1>tokens and learning to predict those masked binary tokens.

Once we have identified the important keywords through our classification process, the next step is to guide the language model in generating the summary. We compared the summary quality based on two dimensions: prompting strategies and decoding strategies.

We explored prompting as the format of (source, [keyword1, keyword2, ...]) in two ways: inclusive and selective.

Inclusive Prompting: 

Figure 4: Inclusive Prompting Example

• We feed the keywords that the model confidently predicts to be <1> into the prompt, instructing the model to include these keywords.

• “You are an assistant summarizing given clinical notes. Ensure to mention the given keywords naturally within the summary.”

Selective Prompting:

Figure 5: Selective Prompting Example

• It may be necessary to exclude certain keywords, and we instruct the model to both include the important keywords and avoid mentioning unnecessary keywords.

• “You are an assistant summarizing given clinical notes. Ensure to mention the given keywords naturally within the summary. Do not include any of the nontarget keywords in the summary.”

Aside from the two prompting strategies, we enhance the factual accuracy of the generated summaries by employing advanced decoding strategies. Since the language model generates text one word at a time—predicting the next word based on the previously generated context—the choice of decoding method can greatly impact the quality of the summary. Simple methods like greedy results in subpar summaries, and even beam search, which evaluates multiple possible sequences simultaneously, may fail to deliver optimal results.

DoLa⁴, which stands for decoding by contrasting layers, is a recent work to mitigate factual inconsistencies in language model generation. This method enhances factuality in both multiple-choice and open-ended generation tasks by contrasting the output probabilities from different transformer layers. Specifically, it treats the output probabilities of the final layers as “expert” predictions while considering those from the middle layers as “amateur” predictions. This approach capitalizes on findings that factual knowledge in large language models is typically concentrated in specific transformer layers, leading to more accurate and reliable text generation.

The first metric for Named Entity labeling are listed as below: