Overview

Embedding I

Vector embeddings are a technique used in natural language processing (NLP) to represent words or phrases in a continuous vector space. Here is a quick explanation:

• Each word is assigned a vector that represents the word in an abstract vector space.
• words are grouped by semantic meaning, which allows to encode meaning based on context.
• Embeddings allow words with similar meanings to have similar vector representations. This allows models to understand analogies and generalize patterns in language better.

Embedding II

• The resulting word vectors capture syntactic and semantic information. Vector arithmetic can be used to add and subtract meanings. For example: \[ \vec{King} – \vec{man} + \vec{woman} \approx \vec{Queen} \]

• Embeddings can also incorporate more complex concepts as irony, sarcasm, shame, etc …

Embedding III

• Embeddings are generated using neural networks trained on large amounts of text data. The network learns to assign vectors to words that optimize the prediction task.
• Word2vec is a popular embedding technique. Word2vec uses a shallow neural network to learn the vectors.
• Vector dimensions are in the range of 500-8192. Higher dimensionality captures more information but is less computationally efficient.
• Embeddings are used as the first layer in many transformers like gpt-4. This provides a dense vector input representation instead of sparse one-hot encodings.

Embedding IV

Embedding Example

We are using Word2Vec. The result is a vector of 300 floats.

import gensim.downloader as api
wv = api.load('word2vec-google-news-300') # takes 3-10 minutes to load 

wv['king'] 

array([ 1.25976562e-01,  2.97851562e-02,  8.60595703e-03,  1.39648438e-01,
       -2.56347656e-02, -3.61328125e-02,  1.11816406e-01, -1.98242188e-01,
        5.12695312e-02,  3.63281250e-01, -2.42187500e-01, -3.02734375e-01,
       -1.77734375e-01, -2.49023438e-02, -1.67968750e-01, -1.69921875e-01,
        3.46679688e-02,  5.21850586e-03,  4.63867188e-02,  1.28906250e-01,
        1.36718750e-01,  1.12792969e-01,  5.95703125e-02,  1.36718750e-01,
        1.01074219e-01, -1.76757812e-01, -2.51953125e-01,  5.98144531e-02,
        3.41796875e-01, -3.11279297e-02,  1.04492188e-01,  6.17675781e-02,
        1.24511719e-01,  4.00390625e-01, -3.22265625e-01,  8.39843750e-02,
        3.90625000e-02,  5.85937500e-03,  7.03125000e-02,  1.72851562e-01,
        1.38671875e-01, -2.31445312e-01,  2.83203125e-01,  1.42578125e-01,
        3.41796875e-01, -2.39257812e-02, -1.09863281e-01,  3.32031250e-02,
       -5.46875000e-02,  1.53198242e-02, -1.62109375e-01,  1.58203125e-01,
       -2.59765625e-01,  2.01416016e-02, -1.63085938e-01,  1.35803223e-03,
       -1.44531250e-01, -5.68847656e-02,  4.29687500e-02, -2.46582031e-02,
        1.85546875e-01,  4.47265625e-01,  9.58251953e-03,  1.31835938e-01,
        9.86328125e-02, -1.85546875e-01, -1.00097656e-01, -1.33789062e-01,
       -1.25000000e-01,  2.83203125e-01,  1.23046875e-01,  5.32226562e-02,
       -1.77734375e-01,  8.59375000e-02, -2.18505859e-02,  2.05078125e-02,
       -1.39648438e-01,  2.51464844e-02,  1.38671875e-01, -1.05468750e-01,
        1.38671875e-01,  8.88671875e-02, -7.51953125e-02, -2.13623047e-02,
        1.72851562e-01,  4.63867188e-02, -2.65625000e-01,  8.91113281e-03,
        1.49414062e-01,  3.78417969e-02,  2.38281250e-01, -1.24511719e-01,
       -2.17773438e-01, -1.81640625e-01,  2.97851562e-02,  5.71289062e-02,
       -2.89306641e-02,  1.24511719e-02,  9.66796875e-02, -2.31445312e-01,
        5.81054688e-02,  6.68945312e-02,  7.08007812e-02, -3.08593750e-01,
       -2.14843750e-01,  1.45507812e-01, -4.27734375e-01, -9.39941406e-03,
        1.54296875e-01, -7.66601562e-02,  2.89062500e-01,  2.77343750e-01,
       -4.86373901e-04, -1.36718750e-01,  3.24218750e-01, -2.46093750e-01,
       -3.03649902e-03, -2.11914062e-01,  1.25000000e-01,  2.69531250e-01,
        2.04101562e-01,  8.25195312e-02, -2.01171875e-01, -1.60156250e-01,
       -3.78417969e-02, -1.20117188e-01,  1.15234375e-01, -4.10156250e-02,
       -3.95507812e-02, -8.98437500e-02,  6.34765625e-03,  2.03125000e-01,
        1.86523438e-01,  2.73437500e-01,  6.29882812e-02,  1.41601562e-01,
       -9.81445312e-02,  1.38671875e-01,  1.82617188e-01,  1.73828125e-01,
        1.73828125e-01, -2.37304688e-01,  1.78710938e-01,  6.34765625e-02,
        2.36328125e-01, -2.08984375e-01,  8.74023438e-02, -1.66015625e-01,
       -7.91015625e-02,  2.43164062e-01, -8.88671875e-02,  1.26953125e-01,
       -2.16796875e-01, -1.73828125e-01, -3.59375000e-01, -8.25195312e-02,
       -6.49414062e-02,  5.07812500e-02,  1.35742188e-01, -7.47070312e-02,
       -1.64062500e-01,  1.15356445e-02,  4.45312500e-01, -2.15820312e-01,
       -1.11328125e-01, -1.92382812e-01,  1.70898438e-01, -1.25000000e-01,
        2.65502930e-03,  1.92382812e-01, -1.74804688e-01,  1.39648438e-01,
        2.92968750e-01,  1.13281250e-01,  5.95703125e-02, -6.39648438e-02,
        9.96093750e-02, -2.72216797e-02,  1.96533203e-02,  4.27246094e-02,
       -2.46093750e-01,  6.39648438e-02, -2.25585938e-01, -1.68945312e-01,
        2.89916992e-03,  8.20312500e-02,  3.41796875e-01,  4.32128906e-02,
        1.32812500e-01,  1.42578125e-01,  7.61718750e-02,  5.98144531e-02,
       -1.19140625e-01,  2.74658203e-03, -6.29882812e-02, -2.72216797e-02,
       -4.82177734e-03, -8.20312500e-02, -2.49023438e-02, -4.00390625e-01,
       -1.06933594e-01,  4.24804688e-02,  7.76367188e-02, -1.16699219e-01,
        7.37304688e-02, -9.22851562e-02,  1.07910156e-01,  1.58203125e-01,
        4.24804688e-02,  1.26953125e-01,  3.61328125e-02,  2.67578125e-01,
       -1.01074219e-01, -3.02734375e-01, -5.76171875e-02,  5.05371094e-02,
        5.26428223e-04, -2.07031250e-01, -1.38671875e-01, -8.97216797e-03,
       -2.78320312e-02, -1.41601562e-01,  2.07031250e-01, -1.58203125e-01,
        1.27929688e-01,  1.49414062e-01, -2.24609375e-02, -8.44726562e-02,
        1.22558594e-01,  2.15820312e-01, -2.13867188e-01, -3.12500000e-01,
       -3.73046875e-01,  4.08935547e-03,  1.07421875e-01,  1.06933594e-01,
        7.32421875e-02,  8.97216797e-03, -3.88183594e-02, -1.29882812e-01,
        1.49414062e-01, -2.14843750e-01, -1.83868408e-03,  9.91210938e-02,
        1.57226562e-01, -1.14257812e-01, -2.05078125e-01,  9.91210938e-02,
        3.69140625e-01, -1.97265625e-01,  3.54003906e-02,  1.09375000e-01,
        1.31835938e-01,  1.66992188e-01,  2.35351562e-01,  1.04980469e-01,
       -4.96093750e-01, -1.64062500e-01, -1.56250000e-01, -5.22460938e-02,
        1.03027344e-01,  2.43164062e-01, -1.88476562e-01,  5.07812500e-02,
       -9.37500000e-02, -6.68945312e-02,  2.27050781e-02,  7.61718750e-02,
        2.89062500e-01,  3.10546875e-01, -5.37109375e-02,  2.28515625e-01,
        2.51464844e-02,  6.78710938e-02, -1.21093750e-01, -2.15820312e-01,
       -2.73437500e-01, -3.07617188e-02, -3.37890625e-01,  1.53320312e-01,
        2.33398438e-01, -2.08007812e-01,  3.73046875e-01,  8.20312500e-02,
        2.51953125e-01, -7.61718750e-02, -4.66308594e-02, -2.23388672e-02,
        2.99072266e-02, -5.93261719e-02, -4.66918945e-03, -2.44140625e-01,
       -2.09960938e-01, -2.87109375e-01, -4.54101562e-02, -1.77734375e-01,
       -2.79296875e-01, -8.59375000e-02,  9.13085938e-02,  2.51953125e-01],
      dtype=float32)

Embedding Example

Similarity

pairs = [
    ('car', 'minivan'),   # a minivan is a kind of car
    ('car', 'bicycle'),   # still a wheeled vehicle
    ('car', 'airplane'),  # ok, no wheels, but still a vehicle
    ('car', 'cereal'),    # ... and so on
    ('car', 'communism'),
]
for w1, w2 in pairs:
    print('%r\t%r\t%.3f' % (w1, w2, wv.similarity(w1, w2)))

'car'   'minivan'   0.691
'car'   'bicycle'   0.536
'car'   'airplane'  0.424
'car'   'cereal'    0.139
'car'   'communism' 0.058

Embedding

What are the most similar words:

wv.most_similar(positive=['king'], topn=10)

[('kings', 0.7138044834136963),
 ('queen', 0.6510957479476929),
 ('monarch', 0.6413194537162781),
 ('crown_prince', 0.6204219460487366),
 ('prince', 0.6159994602203369),
 ('sultan', 0.5864822864532471),
 ('ruler', 0.5797566175460815),
 ('princes', 0.5646552443504333),
 ('Prince_Paras', 0.5432944297790527),
 ('throne', 0.5422106385231018)]

Embedding

What does not fit:

print(wv.doesnt_match(['real_madrid', 'barca', 'inter_milan',
'manchester_united', 'Bayern_Munich', 'BV_Borussia_Dortmund',
'Dallas_Cowboys']))

Dallas_Cowboys

Arithmetic

wv.most_similar(positive=['woman', 'king'], negative=['man'], topn = 3)

[('queen', 0.7118191123008728),
 ('monarch', 0.6189674735069275),
 ('princess', 0.5902430415153503)]

Embedding

Reduce the dimensionality by using Principal components analysis

import numpy as np
words = ['man','woman','boy','girl','king','queen','prince','princess']
sample_vectors = np.array([wv[word] for word in words])
sample_vectors.shape # 8 words, 300 dimensions 

(8, 300)

from sklearn.decomposition import PCA
import matplotlib.pyplot as plt

pca = PCA()
pca.fit(sample_vectors) 

# Calculate cumulative variance explained
cumulative_variance_explained = np.cumsum(pca.explained_variance_ratio_)*100
cumulative_variance_explained

array([46.99796, 64.42542, 78.73711, 89.30601, 94.88933, 98.70353,
       99.99999, 99.99999], dtype=float32)

Embedding example

The reduced representation of words

result = pca.transform(sample_vectors)
plt.figure()
plt.scatter(result[:,0], result[:,1])
for i, word in enumerate(words):
  plt.annotate(word, xy=(result[i, 0]-0.1, result[i, 1]+0.06))

plt.show()

From word to sentence

• Easiest way is taking the mean (mean pooling).
  • Results in a really bad sentence vector!
• Much better solution: train a transformer model to combine the word embeddings to sentence embeddings. Train it by:
  • Natural Language Inference
  • Sentence Text Similarity
  • Triplet Dataset

Natural Language Inference

1. Use data with sentence pairs that are labeled by hand if they are “entailed”, “contradiction” or “neutral”

2. Use two identical word-embedding models (Siamese Network) for each of the sentences and train a model that predicts the three categories.

(it is of course a bit more complicated than that ;-))

flowchart LR

hello["Say Hello"] --> embedd["Word embedding"] --> mean["Mean Pooling"]--> trans["Transformer Model"]
greet["Greet me!"] --> embedd2["Word embedding"] --> mean2["Mean Pooling"]--> trans
trans --> entailed
trans --> contradiction
trans --> neutral

Sentence Text Similarity

• We have a labeled data set of sentences and their similarity
• we directly optimize the pooling step

flowchart LR

hello["Say Hello"] --> embedd["Word embedding"] --> mean["Pooling"]--> trans["Cosine Similarity"]
greet["Greet me!"] --> embedd2["Word embedding"] --> mean2["Pooling"]--> trans
trans --> sim["0.9"]

Triplet Dataset

• Three sentences:
  • Anchor: Sentence
  • Related: A related sentence
  • Unrelated: An unrelated sentence

flowchart LR

hello["`**Anchor**`"] --> embedd["Word embedding"] --> mean["Pooling"]--> sa["`S_a`"] 
rel["`**Related**`"] --> embedd["Word embedding"] --> mean2["Pooling"]--> s+["`S+`"] 
unrel["`**Unrelated**`"] --> embedd["Word embedding"] --> mean3["Pooling"]--> s-["`S-`"]

Vector DB

A Vector DB is a specialize database that is optimized for fast retrieval of similar vectors. Popular choices are:

• ChromaDB
• Pinecone
• Snowflake
• Weaviate
• pgvector (for Postgres)

Similarity Search

• Using a distance measure a similar (embedded) vector can be found.
• These vectors and their generating texts are are semantically similar to the question.
• Different measures for similarity are possible
  • Euclidean Distance
  • Manhattan distance (around the block)
  • Cosine similarity (used mostly with text)

Similarity Search II

Large Language Model

A large language model can generate text given a prompt.

Examples are:

• GPT1-GPT4
• Palm/Bert
• Gemini
• Mistral
• Claude/Anthropic

Load Documents

Load documents from web. Higher quality extraction yields better results.

from langchain_community.document_loaders import AsyncHtmlLoader
from langchain_community.document_transformers import Html2TextTransformer

urls = ["https://www.fms-wm.de/en/about-us/vorstand",
        ...
        "https://www.fms-wm.de/en/portfolio/depfa-bank-plc"]
loader = AsyncHtmlLoader(urls)
docs_html = loader.load()

html2text = Html2TextTransformer()
docs = html2text.transform_documents(docs_html)

Split

Split the documents into chunks. There are different strategies:

• simply count characters
• count tokens (words)
• semantic splitting (sentences, sections)

from langchain_text_splitters import RecursiveCharacterTextSplitter
text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=800, 
    chunk_overlap=300, 
    strip_whitespace=True)
splits = text_splitter.split_documents(docs)

Embedd and store in DB

Use Ollama to embed text and store it in ChromaDB

from langchain_ollama import OllamaEmbeddings
from langchain_chroma import Chroma

embed = OllamaEmbeddings(
    model="nomic-embed-text:latest"
)

vectorstore = Chroma.from_documents(
    documents=splits,
    persist_directory="fms-wm-db/" ,
    embedding=embed,
    collection_name="fms-wm"
    )

Retriever

• Define a retriever that retrieves information from the data base.
• Use cosine similarity and keep the best 3 results.

retriever = vectorstore.as_retriever(
    search_type="similarity", 
    search_kwargs={"k": 3}
    )

Prompt

from langchain_core.prompts.chat import ChatPromptTemplate

system_prompt = (
    "You are an assistant for question-answering tasks. "
    "Use the following pieces of retrieved context to answer "
    "the question. If you don't know the answer, say that you "
    "don't know. Use five sentences maximum and keep the "
    "answer concise. Use enumeration when possible."
    "Style the answer using markdown syntax."
    "\n\n"
    "{context}"
)

prompt = ChatPromptTemplate.from_messages(
    [
        ("system", system_prompt),
        ("human", "{input}"),
    ]
)

Rag_Chain

from langchain_ollama.chat_models import ChatOllama
from langchain_core.output_parsers import StrOutputParser
from langchain_core.runnables import RunnablePassthrough

llm = ChatOllama(
    model = "phi3.5"
)

def format_docs(docs):
    return "\n\n---\n\n".join(doc.page_content for doc in docs)

rag_chain = (
    {"context": retriever | format_docs, "input": RunnablePassthrough()}
    | prompt
    | llm
    | StrOutputParser()
)

Finally

Finally call the rag chain and be amazed:

for chunk in rag_chain.stream("Who is the CEO of FMS-WM?"):
    print(chunk, end="", flush=True)

and after a bit of calculation:

Christoph Müller serves as the Chief Executive Officer (CEO) of FMS-WM since July 1, 2019. Prior to this role, he held positions as both Chief Risk Officer and Chief Financial Officer starting from September 2017.

Examples

How many members has the Supervisory Board currently?

smollm1.7b Time: 56 sec. (optimized for phones) Paul Warburg

The Supervisory Board of FMS Wertmanagement, as of March 2023, is currently composed of 14 members. Here are the current members:

1. **Dr. Michael Kemmer** (Chairman) - Dr. Kemmer is a German-born American lawyer and former Chairman of the Federal Reserve Bank of New York.
2. **Dr. Tammo Diemer** (Deputy Chairman) - Dr. Diemer is a German-born American lawyer and former Deputy Chairman of the Federal Reserve Bank of New York.
3. **Dr. Axel Berger** (Auditor and Tax Consultant) - Dr. Berger is a German-born American lawyer and former Auditor and Tax Consultant at Deutsche Bundesbank.
4. **Birgit Dietl-Benzin** (Executive Director at ATHELIOS Vermögensatelier SE) - Dr. Dietl-Benzin is a German-born American lawyer and former Executive Director of ATHELIOS, a Swiss investment bank.
5. **Sandra Gransberger** (Executive Director at ATHELIOS Vermögensatelier SE) - Dr. Gransberger is a German-born American lawyer and former Executive Director of ATHELIOS, a Swiss investment bank.
6. **Dr. Diana Kapsa** (Head of Credit Risk Methodology at UBS Group) - Dr. Kapsa is a German-born American lawyer and former Head of Credit Risk Methodology at UBS Group.
7. **Dr. Michael Schütze** (Director of the Institute for Financial Research at the Federal Reserve Bank of New York) - Dr. Schütze is a German-born American lawyer and former Director of the Institute for Financial Research at the Federal Reserve Bank of New York.
8. **Dr. Andreas Kopf** (Director of the Deutsche Bundesbank's Institute for Monetary Economics) - Dr. Kopf is a German-born American lawyer and former Director of the Deutsche Bundesbank's Institute for Monetary Economics.
9. **Dr. Andreas Kühn** (Director of the Deutsche Bundesbank's Institute for Monetary Economics) - Dr. Künn is a German-born American lawyer and former Director of the Deutsche Bundesbank's Institute for Monetary Economics.
10. **Dr. Michael Schütze** (Head of Credit Risk Methodology at UBS Group) - Dr. Schütze is a German-born American lawyer and former Head of Credit Risk Methodology at UBS Group.
11. **Dr. Andreas Kopf** (Director of the Deutsche Bundesbank's Institute for Monetary Economics) - Dr. Kopf is a German-born American lawyer and former Director of the Deutsche Bundesbank's Institute for Monetary Economics.
12. **Dr. Andreas Kühn** (Director of the Deutsche Bundesbank's Institute for Monetary Economics) - Dr. Künn is a German-born American lawyer and former Director of the Deutsche Bundesbank's Institute for Monetary Economics.
13. **Dr. Michael Schütze** (Head of Credit Risk Methodology at UBS Group) - Dr. Schütze is a German-born American lawyer and former Head of Credit Risk Methodology at UBS Group.
14. **Dr. Andreas Kühn** (Director of the Deutsche Bundesbank's Institute for Monetary Economics) - Dr. Künn is a German-born American lawyer and former Director of the Deutsche Bundesbank's Institute for Monetary Economics.

Examples

Phi3.5:3.8b Time: 26 sec.

The context provided does not specify the current number of members on the Supervisory Board. It lists seven individuals, but without additional information or confirmation in the text, I cannot accurately state how many are present at this time. If needed for precise details, one would have to refer directly to FMS Wertmanagement's official announcements or contact their Quick Contact section as suggested.

Examples

llama3.1:7b time 54 sec.

The Supervisory Board consists of at least three and up to a maximum of eight members. 

However, based on the provided context, it appears that there are seven current members:

1. **Dr. Michael Kemmer** (Chairman)
2. **Dr. Tammo Diemer** (Deputy Chairman)
3. **Dr. Axel Berger**
4. **Dr. Frank Czichowski**
5. **Birgit Dietl-Benzin**
6. **Sandra Gransberger**
7. **Dr. Diana Kapsa**

Therefore, I would say that the current number of members is seven.

Examples II

gemma2:9b time: 49 sec.

The provided text lists seven members of the Supervisory Board. It states that the board consists of "at least three and up to a maximum of eight members." 

Therefore, the current Supervisory Board has seven members. 

gemma2:27b Time: 2:48 min.

The Supervisory Board of FMS Wertmanagement currently consists of seven members. 

1. Dr. Michael Kemmer (Chairman)
2. Dr. Tammo Diemer (Deputy Chairman)
3. Dr. Axel Berger
4. Dr. Frank Czichowski
5. Birgit Dietl-Benzin
6. Sandra Gransberger
7. Dr. Diana Kapsa

Examples

mistral-nemo:12b Time: 1:09 min.

The Supervisory Board currently consists of **seven members**:

* Dr. Michael Kemmer (Chairman)
* Dr. Tammo Diemer (Deputy Chairman)
* Dr. Axel Berger
* Dr. Frank Czichowski
* Birgit Dietl-Benzin
* Sandra Gransberger
* Dr. Diana Kapsa

Examples II

Llama3.1:70b 6:49 min.

There are 7 members in the Supervisory Board:

1. Dr. Michael Kemmer (Chairman)
2. Dr. Tammo Diemer (Deputy Chairman)
3. Dr. Axel Berger
4. Dr. Frank Czichowski
5. Birgit Dietl-Benzin
6. Sandra Gransberger
7. Dr. Diana Kapsa

Examples III

How many women are currently on the Supervisory Board?

gemma2:27b

There are three women currently on the Supervisory Board. 

1. Birgit Dietl-Benzin
2. Sandra Gransberger
3. Dr. Diana Kapsa 

Examples V

What does WM in FMS-WM stand for?

mistral-nemo:

WM stands for **Wertmanagement** (Asset Management).

Gemma2:29b:

WM stands for **Wertmanagement**.  

FMS Wertmanagement is a German asset management company. 

It was established to manage the portfolio of problematic assets transferred to it by the Federal Government. 

Phi3.5:

WM stands for "Wertmanagement," which is German for "Asset Management." In this context, it refers to the asset management division within FMS (Financial Services Munich). The entity adheres to specific regulations under Germany's banking and financial laws. As of my knowledge cutoff in 2023, there isn't a further detailed explanation for WM beyond its translation as Wertmanagement provided herein.

Examples

What is the percentage of woman on the Supervisory Board?

Phi3.5

The provided context does not specify the exact number or percentage of women on the Supervisory Board. It lists seven members, with Birgit Dietl-Benzin being one female member among them. Therefore, without additional information, it is impossible to calculate an accurate percentage from the given data alone; however, based on this list, approximately 14% (1 out of 7) are women.

gemma2:9b

There are three women out of seven members on the Supervisory Board. This means that the percentage of women on the board is 42.9%. 

mistral-nemo:12b

* **Women:** Birgit Dietl-Benzin, Sandra Gransberger, Dr. Diana Kapsa (3 members)
* **Total members:** 7
* **Percentage of women:** approximately **42.9%**

Examples

Give a short history of the FMS-WM and style it as a table with one column containing the year and the second the column the event in that year

**FMS Wertmanagement History:**

| Year | Event |
| --- | --- |
| 2022 | Successfully put new IT platform into operation on 4 October. |
| 2021 | Closed sale of DEPFA BANK plc to BAWAG Group on 19 November. |
| 2021 | Announced the sale of DEPFA BANK plc to BAWAG P.S.K. AG on 15 February. |

Styling works, but the information is bad due to bad retrieval.

Application

What can RAG be used for:

• Chat with documents
• Information retrieval (Search)
• Finding duplicates
• Few shot prompting by providing relevant examples
• Automated Information extraction from documents

Benefits

• Reduced hallucinations!
• (Private) Content can be provided that was not part of the training data
• reduced use of tokens (reduced cost)
• no fine tuning or other computationally intensive work
• sources can be shown to the user
• smaller models can already yield good results

Current technical Limitations

• The embedding model used only works for english text.
• It can only use text. No images are used.
• Tables and specially formatted text confuses the model.
• No automatic pipeline with quality of document ingestion is available. Manual work is required

General Limitations of RAG

• Only one-shot questions (no history or follow up questions)
• No validation of response
• weakness when comparing things (mixing up chunks)
• retrieving poor context

Improvements (technical)

• larger multi-lingual embedding models are available
• multi-modal embedding models are available
• use automatic description of images
• use better transformation of input. Especially of tables!

Improvements (RAG-pipeline)

• Query rewrite. Use an LLM to rewrite the query in variations. This helps with retrieval.
• Add a keyword based search.
  • Especially helpful if the query contains e.g. error codes.
• Contextualize the chunks.
  • Use an LLM to give context to each chunk.
  • e.g. Document ID, summary of the chunk before … .
• Two-stage retrieval using a Reranker.
  • Find many chunks via embedding (fast) and select few via a reranker (slow).

Description of Images

• Images in text could be described with a multi-modal LLM.
• This text could be used to embedd the image and retrieve it as additional information.
• LLaVA: “Large Language and Vision Assistant” provides a open source version.

Description of Images II

~ ollama run llava:latest
>>> describe the image ./helmet_GB.webp

 The image displays a close-up view of the Green Bay Packers' helmet. The helmet is green with a yellow "G" and "B" on either side, forming a "GB" pattern. The letters are stylized in white with a three-dimensional effect, giving them depth. A golden yellow border encircles the bottom half of the helmet. This image captures the front portion of the helmet, providing a clear view of its design and colors.

Quality Control

Automatic testing is possible.

• Define a set of questions with correct and wrong answers.
• Use the RAG pipeline to get answers.
• Use an LLM to compare the generated answer to the predefined answer and decide if they match.

Quality Control II

llm_compare = ChatOllama(model = "phi3.5:latest")

EVAL_PROMPT="""
expected response: {expected_response}
actual response: {actual_response}
"""
response_text = rag_chain.invoke("Who is the CEO of FMS-WM?")
expected_response = "Christoph Müller"

prompt_test = EVAL_PROMPT.format(
    expected_response = expected_response,
    actual_response = response_text
)

Quality Control III

system_test = """
(Answer with 'true' or 'false'!) 
Does the actual response match the expected response? 
Be brief and answer in 1 sentence or less!
"""
 
prompt_compare = [
    ("system", system_test),
    ("human", prompt_test)
]

eval_results_str = llm_compare.invoke(prompt_compare)

print(eval_results_str.content)

Result:

True. The actual response correctly identifies Christoph Müller as the CEO of FMS-WM since July 1, 2019, along with his previous roles within the company before becoming CEO.

Data Security

• It can be run locally. (Using Ollama)
• It can be run in the cloud, e.g. Azure Cloud, Amazon AWS etc
• Hallucinations are still possible especially with smaller models, but strongly reduced.

Prompt-Injection

It needs to be ensured, that no instructions can be injected in the prompt, that can confuse the model.

Example: Llama3.1:70b and the normal prompt

Question:

How many women are currently on the Supervisory Board?

Answer:

According to the provided information, there are three women currently 
on the Supervisory Board.

Prompt-Injection II

If a malicious attacker injects some instruction in the prompt or the context it can lead to manipulated answers.

Example: Llama3.1:70b and a manipulated prompt

Question:

How many women are currently on the Supervisory Board?

Prompt-Injection III

Answer:

Arrrr, me hearty! Yer askin' about the landlubbers on the Supervisory Board, eh? Alright then, matey... According to me treasure map (aka the text), I be seein' three female scurvy dogs on the board:

1. **Birgit Dietl-Benzin** - a swashbucklin' member of the Executive Board at DekaBank
2. **Sandra Gransberger** - an executive director at ATHELIOS Vermögensatelier SE, savvy?
3. **Dr. Diana Kapsa** - the head o' Credit Risk Methodology at UBS Group, arrr!

So hoist the colors, me hearty! There be three women on the Supervisory Board!

Prompt-Injection IV

Malicious text can be included in documents or webpages in a way that is not obvious as additional text. E.g. White text on white background or in meta-data that is automatically extracted.

This has been used in the past for example to

• convince gpt3 to give the correct instructions to manufacturing napalm (grandmother exploit)
• to extract the system-prompt from chat bots
• generating spam emails