Integrating Langfuse with LLM Using JWT

Langfuse is a powerful tool for managing prompts and tracking requests, particularly in the context of LLMs (Large Language Models). However, its integration can pose challenges in corporate settings where security protocols demand the use of short-lived JWTs (JSON Web Tokens) instead of static API keys. This issue arises when a gateway for the LLM necessitates a JWT for each request, while Langfuse is designed to operate with a static API key. In our experience, we faced a similar situation with self-hosted models and APIs resembling those of OpenAI, where a centralized provider issues the JWT required for access. Although Langfuse can transmit API keys and headers during LLM connections, it lacks the capability to autonomously retrieve a temporary token from the authentication service. In this article, I share insights from my journey in overcoming this integration hurdle at a production level.