How to Use LLMs from the Hugging Face Inference API

Introduction

The Hugging Face Inference API is a powerful service that lets you interact with large language models (LLMs) hosted on the Hugging Face Hub. Whether you’re building chatbots, summarization tools, or other AI-powered applications, the Inference API makes it simple to send prompts to an LLM and receive structured, high-quality responses.

In this tutorial, you’ll learn how to use the Hugging Face Inference API in Python. We’ll walk through storing your API key securely, setting up the client, and making your first request to an LLM. For demonstration, we’ll use the meta-llama/Llama-3.1-8B-Instruct model, but the steps apply to any LLM available on the Hugging Face Hub.

🔍 What is the Hugging Face Inference API?

The Hugging Face Inference API is a cloud-based service that allows developers to run machine learning models—especially large language models—without setting up their own servers or GPUs. With just an API key and a few lines of code, you can send text (or other data) to a hosted model and receive a structured response.

Here’s why the Inference API is so useful:
✅ No infrastructure required: Skip the hassle of managing GPU clusters or scaling servers.
✅ Easy model switching: Use any compatible model from the Hugging Face Hub by simply specifying its name.
✅ Low latency: Benefit from optimized serving infrastructure that returns responses quickly.
✅ Provider flexibility: Some models support multiple inference providers so you can choose the one that best suits your needs.

✅ Prerequisites

Before you start, make sure you have:

• Python 3.10+ installed
• A working Python environment
• An access token from your Hugging Face account

🛠️1️⃣ Install Libraries

First, install the Python packages huggingface_hub and python-dotenv.

pip install huggingface_hub python-dotenv

This will also install any required dependencies needed to interact with the Inference API.

📦2️⃣ Import Packages

Now, let’s import the necessary packages in your Python script:

import os
from dotenv import load_dotenv
from huggingface_hub import InferenceClient

This ensures you have everything ready to work with environment variables and the Inference API.

Learn more about how to implement an end-to-end AI solution in our newly published book. We guide you step-by-step through the implementation by using the data platform Microsoft Fabric. Whether you’re a beginner or looking to deepen your expertise, you’ll gain practical insights and hands-on techniques to successfully deploy AI in real-world scenarios.

To the Book

🔑3️⃣ Store Your API Key

To use the Hugging Face Inference API, you need to pass your access token as the API key. It’s important to keep the access token safe and out of version control. A recommended way to manage it locally is by using a .env file and the python-dotenv package.

Create a file named .env in the root of your project and add your token:

HF_API_TOKEN=hf_xxxxxxxxxxxxxxxxxxxxxxxx

In your Python code, load the key:

load_dotenv()
api_key = os.getenv("HF_API_TOKEN")

🚀4️⃣ Initialize the Inference Client

Next, initialize the InferenceClient with your API key and select the desired inference provider: