Fordel StudiosTest infrastructure that keeps pace with Cursor-speed development.
The flaky test problem is a symptom of test infrastructure that was not built for the pace AI-assisted development creates. We build QA pipelines where AI generates test scaffolding, Playwright selectors self-heal on DOM changes, and your LLM features have evaluation harnesses — not just hope — behind them.
Teams shipping with Cursor and Copilot report a consistent pattern: code output increases, test coverage does not. The tools generate the implementation but rarely generate the tests that validate it. When tests do get generated, they often test the implementation rather than the specification — a pattern called confirmation bias testing that looks like coverage but provides no safety net against incorrect behavior.
There is a second problem specific to AI features: non-deterministic LLM outputs cannot be validated with exact-match assertions. A customer support agent that was working last week and was subtly degraded by a model update or a prompt change will not fail a unit test. It will just start giving slightly worse answers until someone notices. Without an eval harness — a dataset of representative inputs with scoring criteria — you have no signal on whether your LLM features are regressing.
| QA challenge | Traditional approach | AI-augmented approach |
|---|---|---|
| Test generation speed | Manual — developer writes every test | LLM generates scaffolding from types and signatures; developer reviews |
| E2E selector maintenance | Manual updates every UI change | Playwright semantic selectors plus self-healing on DOM changes |
| Visual regression | Manual screenshot comparison | Chromatic or Percy with AI-assisted noise filtering |
| LLM output quality | No standard mechanism | LangSmith eval datasets with LLM-as-judge scoring |
| Flaky tests | Manual investigation per flake | Retry analysis and selector stability scoring to surface root cause |
We build test infrastructure at all three layers of the testing pyramid. Fast unit tests with real coverage at the base — generated scaffolding plus human-reviewed assertions. Integration tests for service boundaries and external API behavior. A focused Playwright E2E suite for critical user journeys, using ARIA roles and data-testid selectors rather than CSS paths that break on every redesign.
For AI features, the eval harness is the most important investment. We build LangSmith evaluation datasets with representative inputs and scoring criteria: rule-based checks for structured outputs, semantic similarity for prose, and LLM-as-judge for subjective quality dimensions. Aggregate scores are tracked over time. A model update or prompt change that degrades performance shows up in the eval run before it reaches users.
AI-powered QA layers we build
LLM generates test scaffolding from function signatures and types as part of the PR flow. Generated tests are flagged for human review on assertion logic. Coverage gaps are reported automatically. Shift-left: quality feedback in the PR, not post-merge.
Playwright tests written against semantic selectors — ARIA roles, labels, data-testid attributes — that survive CSS and layout changes. When selectors break, the system attempts automated repair. Critical path coverage: auth, core value delivery, payment flows.
Chromatic captures component and page screenshots on every PR. Pixel diffs are reviewed in the Chromatic UI with baseline management. Storybook integration means component-level visual regression runs alongside unit tests.
LangSmith evaluation datasets with representative inputs, expected behavior criteria, and automated scoring pipelines. LLM-as-judge scoring for subjective quality. Ground-truth comparison for structured outputs. Regression alerts when scores drop.
LLM-assisted test scaffolding
We integrate LLM-powered test generation into the development workflow: scaffolding is generated from function signatures and types, developers review and complete assertion logic. The goal is to eliminate the time cost of test setup, not the judgment cost of test design.
Playwright E2E with semantic selectors
Playwright is the current standard for reliable browser automation. We write tests against ARIA roles and data-testid attributes rather than fragile CSS selectors. Tests survive component redesigns. Self-healing reduces the maintenance burden when selectors do break.
LangSmith evaluation harnesses
We build evaluation pipelines for AI features with ground-truth datasets and automated scoring. LLM-as-judge scoring for subjective quality dimensions. Aggregate pass rates tracked over time. This is how you detect regressions in AI feature quality.
Visual regression with Chromatic
Chromatic integrates with Storybook and Next.js to capture visual snapshots on every PR. Component-level and page-level regression detection with baseline management and review workflow built in.
Shift-left quality gates
Quality checks run as close to the developer as possible: pre-commit linting, PR-level test runs, automated coverage reporting, and eval harness runs on changes to AI-adjacent code. Issues caught in the PR are cheaper to fix than issues caught post-merge.
- Test coverage audit with gap analysis across unit, integration, and E2E layers
- AI-assisted test generation pipeline integrated into PR workflow with human review gates
- Playwright E2E suite for critical user journeys with self-healing semantic selectors
- Chromatic or Percy visual regression pipeline with noise-filtering configuration
- LangSmith evaluation harness with datasets and scoring pipelines for LLM features
- CI quality gates with coverage thresholds, flake detection, and regression blocking
Teams with eval harnesses for their LLM features ship model and prompt updates with confidence. Teams without them discover regressions from user complaints. The Playwright and visual regression infrastructure reduces the maintenance burden that kills E2E suites at scale.
Common questions about this service.
How do you test non-deterministic LLM outputs?
Not with exact-match assertions. We build evaluation datasets with representative inputs and expected output characteristics — not exact strings. Scoring uses rule-based checks for structured fields, semantic similarity for prose, and LLM-as-judge for subjective quality. Aggregate pass rates over the dataset are tracked over time via LangSmith. A drop in aggregate score is the regression signal.
Do you work with existing test suites?
Yes. We audit existing coverage, identify high-value gaps, and extend rather than replace. Complete rewrites are rarely justified — the value is adding the AI-specific eval infrastructure that existing suites cannot provide, and filling coverage gaps in areas where the test/code ratio has degraded.
What is the flaky test problem and how do you address it?
Flaky tests — tests that pass and fail intermittently without code changes — are a signal of unstable selectors, timing dependencies, or test isolation failures. We address root causes: replace CSS selectors with semantic ones, add proper async waiting in Playwright, isolate test data between runs. Chromatic's flake detection and Playwright's retry analysis help surface which tests are flaky and why.
What is the right E2E coverage scope?
E2E tests are expensive to maintain. The target is not 100% coverage — it is coverage of the flows where a failure would be immediately visible to users and high-impact: authentication, core value delivery, and payment/subscription flows. Everything else is better covered at unit and integration layer.
Ready to get started?
Tell us what you are building. We will scope it, price it honestly, and give you a clear plan.
Start a ConversationFree 30-minute scoping call. No obligation.