Walkthrough: Integrating Perplexity API with our internal knowledge base.

Alright, let's get this over with. Another "revolutionary" API that promises to magically understand your internal docs. Spoiler: it's mostly about chunking, embeddings, and RAG like everything else these days. But the team pushed for a POC, so I integrated the Perplexity API with our internal Confluence/KG dump. Here's the raw, unfiltered walkthrough of what actually worked, where it choked, and the config that didn't explode.

First, the architecture—because they never show you the plumbing. We're not using their chat completion endpoint for simple Q&A; we're using the `sonar-reasoning` model for the actual retrieval and reasoning, and `sonar` for the online search when needed. The flow is standard:

* Documents are chunked, embedded (we kept using our existing `text-embedding-3-small` pipeline because it's cheap and we already track the drift), and stored in PGVector.
* On query, we pull the top-k relevant chunks via cosine similarity.
* Those chunks, plus the query, get stuffed into a system prompt for the Perplexity API.
* We force it to cite from the provided context, and fall back to online search only if the user explicitly requests it or confidence is below a threshold.

The real meat is in the prompt engineering and the cost controls. Without those, you'll burn credits on hallucinations and get answers that sound confident but are utterly wrong.

Here's the core system prompt we landed on after about two hundred test queries (and fifty bucks in API spend):

```text
You are an internal assistant for a data engineering team. Use ONLY the provided context documents to answer the user's technical question. The context is from our internal knowledge base, Confluence pages, and runbooks.

STRICT RULES:
1. If the answer is not contained within the provided context, respond with: "Based on our internal documentation, I cannot provide a specific answer to this question."
2. Do not speculate or combine prior knowledge with the context.
3. Cite the relevant document IDs (e.g., [doc: 12]) for each statement you make.
4. If the user's question is ambiguous within the context, ask for clarification.

Provided Context:
{context_str}

User Question: {query}
```
The key was rule #1. Without it, the model would happily ignore our carefully retrieved chunks and start hallucinating procedures based on its general training. We also had to index the chunk IDs alongside the embeddings for citation.

Now, the actual API call structure (Python, because the DevOps folks insisted on a FastAPI wrapper):

```python
import os
from openai import OpenAI

client = OpenAI(
api_key=os.environ["PPLX_API_KEY"],
base_url="https://api.perplexity.ai"
)

def query_internal_knowledge(question: str, context_chunks: list[str]):
context_str = "n---n".join(context_chunks)
system_prompt = # ... see above, formatted with context_str and question

response = client.chat.completions.create(
model="sonar-reasoning",
messages=[
{"role": "system", "content": system_prompt},
{"role": "user", "content": question}
],
temperature=0.1,
max_completion_tokens=1024,
)
return response.choices[0].message.content
```

Observations and Pitfalls:

* **Cost:** The `sonar-reasoning` model is not cheap. It's more expensive than GPT-4 Turbo for our use case. We're only using it for the final reasoning step; the heavy lifting of retrieval is still our old, cheap pipeline.
* **Speed:** It's slower than a direct `gpt-4` call, noticeably so when you have a chain of dependent questions.
* **Search Integration:** Their "online search" feature via the `sonar` models is neat, but we had to gate it heavily. You don't want your internal assistant leaking internal schema names or table structures in a web search. We only allow it for generic, non-sensitive tooling questions (e.g., "what's the latest version of Apache Flink?").
* **The "Pro" models:** They push you towards `sonar-reasoning-pro` and such. In our testing, the accuracy gain for straightforward technical documentation Q&A was marginal—not worth the 2x cost bump.

In the end, it works. It's marginally better at parsing poorly written old runbooks than our previous fine-tuned `gpt-3.5-turbo` setup, but I'm not convinced the delta justifies the vendor lock-in and the added latency. The main win was the team finally agreeing on a strict RAG pattern with citations. We could probably port this to any other model endpoint next quarter when the next shiny thing comes along.

-- old salt