All projects

Building LLM Observability

April 23, 2024/ Founding Designer
/ FigmaHTML/CSS/JSCursor
Building LLM Observability

Helicone (YC W23) launched as the all-in-one LLM observability platform for developers, focused primarily on individual developers and AI startups. The initial interface spoke "developer" fluently but lacked the visual sophistication enterprise customers look for when evaluating mission-critical AI toolings.

Philosophy

ship fast, ship often

The 'AHA' Moment

Developer-first = design for productivity

Before Helicone, when something went wrong with our AI features, we'd spend hours trying to figure out which requests failed and why. It was like searching for a needle in a haystack. - Engineering Lead at an AI-first startup

Tools Used:

  • Figma
  • HTML/CSS/JS
  • Cursor

Request Page Redesign

Status: Live on Helicone's Request Page

Role: User Research, Design

Helicone Request Page

The Challenge

The request view—where users could see and analyze individual LLM interactions—was at the heart of this problem. Our initial implementation provided basic functionality but failed to address several crucial needs:

User Research Findings

  • 78% reported troubleshooting LLM issues took 3+ hours on average
  • 92% had experienced unexpected LLM behavior in production that impacted users
  • 63% were manually logging requests to understand patterns
  • 85% couldn't effectively monitor costs and performance in real-time

These conversations revealed that the request view wasn't just a monitoring feature—it was the primary interface through which teams understood their LLM applications' behavior.

  1. Real-time visibility: Users couldn't easily see requests as they happened
  2. Context preservation: Correlating prompts with their responses was difficult
  3. Metadata insights: Critical LLM-specific data was hard to interpret
  4. Pattern recognition: Identifying problematic trends required manual effort
  5. Quick troubleshooting: Finding and diagnosing issues took too long

Sessions Redesign

Research Findings

Impact

The visual transformation moved away from the feature-first approach towards a confidence-signalling aesthetic that enterprise buyers resonated with.

The cleaner interface drew immediate positive feedback, while outreach from companies like Roblox and Singapore Airlines confirmed that thoughtful visual design can dramatically reposition a technical product in the B2B space.

Learnings

Quotes

Curious to learn more? Reach out to me.