Abdur Rehman

What it is

A voice assistant you speak to right in the browser. It answers as Abdur Rehman Afzal and grounds every reply in his real portfolio using retrieval (RAG), so it stays factual instead of making things up.

How it works — the pipeline

Your voice streams over WebRTC through a cascaded, low-latency pipeline:

1. Silero VAD detects when you start and stop speaking, and lets you interrupt.
2. Deepgram turns your speech into text.
3. OpenAI gpt-4o-mini writes the reply, and can call a search tool.
4. Cartesia turns that reply into natural speech.
5. The audio streams back to your browser.

Grounded answers (RAG)

Rather than guessing, the model calls a search_portfolio tool. The knowledge base is embedded once with sentence-transformers (all-MiniLM-L6-v2); each question is matched by cosine similarity and the top results are fed back, keeping answers accurate. It is real, on-demand retrieval plus agentic tool use.

Tech stack & tools

Orchestration

Pipecat (Python)

Speech-to-Text

Deepgram

Language model

OpenAI gpt-4o-mini

Text-to-Speech

Cartesia

Turn-taking

Silero VAD

Transport

SmallWebRTC (peer-to-peer, no media server)

Retrieval

sentence-transformers + NumPy cosine similarity

Server

FastAPI + Uvicorn

Frontend

Vanilla JS + Pipecat JS SDK

What it demonstrates

Real-time voice AI, agentic tool-use, retrieval-augmented generation, low-latency pipeline design, cost controls (session caps and opt-in audio), and single-service deployment behind HTTPS.

Build your own — a quick guide

1. Get one voice loop working with Pipecat's quickstart (STT → LLM → TTS).
2. Give it a persona through the system prompt.
3. Ground it: embed a knowledge base and expose retrieval as a tool.
4. Brand the UI and connect it over WebRTC.
5. Add cost guards and deploy it behind your own domain.