Redesigning LLM Observability for Developers

Challenge

Designing for Developers = Designing for Productivity

I joined Helicone (YC W23) as a founding designer to reimagine the core experience of LLM observability from scratch. As the company grew, we needed to redesign the core platform to support enterprise clients.

Product & GTM Impact

After the redesign, we saw a consistent 10-20% MoM growth in signups, a 3.6% increase in free to paid conversion, and doubled organic traffic in one week through content strategy.

Philosophy

Prioritize, Ship Fast, Ship Often

We anchored our design strategy around three core principles to accelerate user value and business outcomes:

Request Redesign

Having Strong Opinions About the Core Information We Show

Problem

As the product grew, the complexity was working against developer productivity rather than supporting it.

The request page is Helicone's most visited page —live in product—where users can see and analyze individual LLM interactions. I focused the redesign on surfacing only the most essential debugging information, cutting noise and aligning with real usage patterns.

A simple audit reveals that the drawer needed to surface the right information without overwhelming users, and look consistent with the rest of the product.

How might we display the most critical data for developers who need quick observability answers?

After speaking to 5 companies with top usages of Helicone, I found that:

Solution

Defaultly Hide Metadata, Shortcuts to Session/Prompts

By hiding metadata by default and simplifying the drawer, we helped developers cut time-to-insight by 60% and focus immediately on key debugging info - the request and response pair.

Edge Cases

While most users log only LLM calls today, the future of AI applications is multimodal — increasingly dependent on tool and vector calls.

LLM calls focus on prompt/response pairs, while tool calls need parameter validation and vector queries require similarity scoring visualization. Thus they required a different type of visualization than LLM calls.

Interactions

By default, metadata is hidden so developers can focus on request/response bodies, with full details accessible when needed. I designed this progressive disclosure approach to match how developers actually debug: as they scroll through responses, non-essential UI elements fade away, keeping focus on the core data.

Accessibility

Since most developers work in dark-themed environments (VS Code, terminals, dashboards), dark mode was essential for Helicone to fit seamlessly into their workflows without jarring visual interruptions that bright interfaces create.

Sessions Redesign

Help Developers Pinpoint Errors in Complex LLM Workflows

With the request page simplified, I redesigned the Sessions view to solve our second biggest user pain point.

User Research

90% of users follow this debugging workflow: pinpoint errors, understand surrounding context, aggregate metrics for deeper LLM application analysis.

This revealed that users spend most of their time on steps 1 and 2, which we were well-positioned to optimize. Our interface needed to be opinionated about surfacing key insights upfront, answering critical questions like "What's the average session cost?" and "What's the typical latency?" without forcing developers to dig for answers.

How might we guide developers to the root cause error in their LLM workflows containing many sequential requests?

Before

Insights

What questions do developers want to ask their LLM data?

While engineering wanted a list of bug reports and performance improvements, from a product/marketing perspective, we want to allow developers to navigate from a request to Sessions easily in order to upsell on advanced features. In order to create a smooth upsell experience, we needed to ensure developers get the most value out of the Sessions page.

Solution

Error Indicator, Aggregate Metrics to Answer Questions Easily

"It's nice to be able to collapse everything and then determine, 'Okay, we think the error is in this part,' and expand that and look there. Sessions really helps us locate where the issue is using the paths."

Founding Engineer at Tusk

Filter Improvement

78% of debugging time was spent not on fixing issues, but on simply finding the right request.

This insight shifted our design from better analytics to better filtering, designed to help developers find the right request faster and ultimately reducing time-to-insight by 60%.

Top-of-Funnel

Leading Developer-First Content Strategy

Mapping out the user journey, I found that developers at different stages of their journey with Helicone need different support. The most pressing need was to increase awareness of the product.

Designing a Scalable Content System to Drive Conversion

Helicone was winning word-of-mouth in the dev community, but our traffic did not convert into signups. After auditing 50+ dev-first content sites (e.g. Vercel, Supabase, LangChain), I created a modular content system that allowed us to create consistent, high-quality content that converts.

Capturing Customers at Scale with Engineering-as-Marketing Tools

We built high-utility tools like LLM comparison tool, API cost calculator, open stats and provider status page, turning fun engineering pages into scalable marketing assets that brings in qualified traffic. This approach helped us establish trust with technical audiences, and drive top-of-funnel growth organically.

Impact

Our content strategy helped Helicone 2x traffic MoM, with a 3.9% conversion rate on average, and helped Helicone become the most visible brand in AI search.

Key Learnings

Next Steps

While the redesign achieved significant improvements in developer productivity, the next phase focuses on systematic testing and measurement to drive continuous iteration.

1. Watch session recordings on the request drawer and sessions to validate that users actually spend 90% of time on request/response bodies.
2. Track average time from landing on request page to finding target error request.
3. Conduct monthly interviews with 5-8 power users to track workflow evolution.

Curious to learn more? Reach out to me.