Introduction
This post will walk you through building a text inference application using Meta’s Llama-3.1 model. We will use the data framework LlamaIndex python package in our app. I’m running this demo on WSL2 (Windows Subsystem for Linux) on Windows 11. Python 🐍 is run in an ipynb file on Visual Studio Code for Linux. I’ll try to setup a github repo for this as well.
Setup
You’ll need a hugging face account and at least a read only token to obtain the models (Should be free).
Your PC should have a decent GPU and 32GB of memory. I was able to run this on a 16GB machine as well but it’s really pushing it.
Also get permission to use Meta’s Llama-3.1 models on hugging-face.
This is how I logged into huggingface: huggingface-cli login –token <your token here>
Save your token as you’ll need it to log in again next time.
pip install -U torch
pip install -U llama-index-core
pip install transformers bitsandbytes llama-index-readers-file llama-index-llms-huggingface llama-index-embeddings-huggingface
nvidia setup:
Here are the packages after I install torch, notice it picks up cuda support.
Building the app
Import required packages:
Configure the settings:
We will use the instruction tuned version of the 8B model and also select an embedding model.
Store the index in a storage directory. You can see how handy the LlamaIndex package is for geting and preparing the data. For data I used a text file with contents being the book “Anne of Green Gables” taken from the web, utf-8 encoding.
Now run text query(s):
Output:
Question: How did Anne cope?
response=’ The text does not provide a direct answer to this query. However, based on the context information, it can be inferred that Anne passed her examinations and was thrilled about it. She felt a sense of accomplishment and pride. She also felt a sense of joy and excitement when she learned that she had passed, as evident from her reaction when Diana broke the news to her. This suggests that Anne was able to cope with the pressure and stress of the examinations by staying focused and motivated. She also seems to have a positive and optimistic outlook on life, as evident from her imaginative and creative nature. Overall, Anne appeared to cope well with the challenges she faced.’