Commit Graph

257 Commits

Author SHA1 Message Date
Brian Hackett
3c3a97aa50
Show information about anthropic calls when clicking usage (#34) 2025-02-19 16:19:42 -08:00
Chris Toshok
4795d02150
instrument whether an api key is the user's or ours (#33)
introduce an `AnthropicApiKey` type so we aren't passing a string/boolean everywhere.
2025-02-19 13:58:40 -08:00
Chris Toshok
8d3a8eeb3a
use AsyncLocalStorageContextManager to propagate trace context around (#32)
I noticed that the `llm.chat.*_tokens` attributes weren't in spans, and it turns out none of the attributes from the wrapped function make it in.  only those added synchronously from within `wrapWithSpan` do.

A few things in play here:

1. Switch from `ZoneContextManager` (which doesn't work with native async/await) to `AsyncLocalStorageContextManager` (which does.)
2. (1) requires we use a pnpm patch, since wrangler requires that we import `async_hooks` statically, and the otel code `requires` it, and also requires that it's spelled `node:async_hooks`.
3. encode the payload we try to send _before_ doing the retry loop, so we don't re-encode for each attempt.
4. switch from `startActiveSpan` to `context.with` + `startSpan`.  The latter allows us to wrap an async function.  the former is for synchronous functions.
2025-02-19 06:49:32 -08:00
Chris Toshok
c52ae1f9da
Fix otel errors (#31)
Running the prior code in wrangler (but _not_ `npm run dev`) was generating errors during otel initialization.  I was ultimately unable to figure out why, or how to fix them:

```
  (warn) Initializing OpenTelemetry
  (error) Error initializing OpenTelemetry TypeError: Class constructor OTLPExporterBase2 cannot be invoked without 'new'
```

In the end, it was easier to just boil the ocean a little:  With claude's help, write a custom otlp exporter and use that.  Also use BasicTracerProvider instead of WebTracerProvider.

This version is enough to generate spans from within local wrangler.
2025-02-18 16:18:03 -08:00
Brian Hackett
78a56faa55 Followup fixes 2025-02-18 15:16:33 -08:00
Brian Hackett
55dd3c97b6
Add feedback button (#29) 2025-02-18 15:15:52 -08:00
Brian Hackett
76059dc19c otel changes 2025-02-18 12:40:44 -08:00
Brian Hackett
c00ad7792c more logging 2025-02-18 12:16:01 -08:00
Brian Hackett
ab614ee90a Telemetry diagnostics 2025-02-18 12:12:12 -08:00
Chris Toshok
d81ab41688
warn that otel wasn't initialized properly, every time we would have sent a span (#30)
The log should contain a warning showing the following once at startup:
```
OpenTelemetry initialization skipped: HONEYCOMB_API_KEY and/or HONEYCOMB_DATASET not set
```
and every time a span would have been created:
```
OpenTelemetry not initialized, skipping span creation
```

Also sneak in a `service.version` attribute that is `${__APP_VERSION}; ${__COMMIT_HASH}`.  e.g.:
![image](https://github.com/user-attachments/assets/6f111cb1-8fb9-46c4-9bd8-24bdd08650ab)
2025-02-18 12:00:38 -08:00
Brian Hackett
6221ab1d3e
Fix issues around rewinding (#28) 2025-02-18 11:39:44 -08:00
Chris Toshok
af4c405620
[PRO-973] basic o11y around anthropic api calls (#22)
add some otel initialization, pointing at honeycomb + a function to wrap other functions with spans.  use that function to wrap `callAnthropic` and add some context (number of messages sent, token usage, along with some metadata should we ever support other providers.)

The honeycomb api key/dataset are to be passed via env vars `HONEYCOMB_API_KEY` and `HONEYCOMB_DATASET` respectively.
2025-02-18 10:06:20 -08:00
Brian Hackett
20bb97c0c5
Fix assorted problems (#27) 2025-02-14 14:27:20 -08:00
Brian Hackett
b2efa3b56d Fixes for partial update handling 2025-02-13 16:34:30 -08:00
Brian Hackett
9167d77ff2
Save scrolling interactions (#26) 2025-02-13 14:31:03 -08:00
Brian Hackett
2f2a239660
Avoid losing code when prompting AI (#24) 2025-02-13 09:36:22 -08:00
Brian Hackett
dd18d5360c
Support rewinding to earlier messages (#23) 2025-02-12 17:23:26 -08:00
Brian Hackett
43ab9b3656 Support saving solutions without an associated evaluator 2025-02-12 14:19:57 -08:00
Brian Hackett
4cf70843c0
Add UI for updating problems with admin key set (#18) 2025-02-12 08:59:42 -08:00
Brian Hackett
b2a98b4180 tweak logging 2025-02-11 17:15:00 -08:00
Brian Hackett
d80eff60a7 Handle older format for repository contents in problems 2025-02-11 13:42:12 -08:00
Brian Hackett
fc89c55517 Remove obsolete experimental command 2025-02-11 09:57:33 -08:00
Brian Hackett
ce94bf7fcf Include chat messages in solution instead of last user prompt 2025-02-11 06:07:57 -08:00
Brian Hackett
bb82c56958
Improve UI for viewing and changing problems (#15) 2025-02-10 20:18:54 -08:00
Brian Hackett
b7b602016e
Improve support for streaming simulation data to backend (#16) 2025-02-07 11:52:19 -08:00
Mateusz Burzyński
d143863285
Record pointer moves (#14) 2025-02-07 15:37:39 +01:00
Mateusz Burzyński
ce1eed19bb
Include viewport size in the simulation data (#13)
* Include viewport size in the simulation data

* add reusable type
2025-02-06 17:02:33 +01:00
Mateusz Burzyński
aec72dd575
More resilient preview script injection and postMessage communication (#12) 2025-02-04 20:45:18 +01:00
Brian Hackett
547b027ef7 Log all simulation data in console 2025-02-03 20:28:03 -08:00
Brian Hackett
3c193be3e6 Include mouse data when enhancing prompt 2025-01-30 11:24:58 -08:00
Brian Hackett
df8e2526ee
Use Nut Chat API (#11) 2025-01-29 10:45:18 -08:00
Brian Hackett
45c7365f01 Include mouse data when analyzing execution data 2025-01-23 07:35:06 -08:00
Brian Hackett
7d02bb58fc Ignore package-lock.json 2025-01-22 15:26:40 -08:00
Brian Hackett
7ce6110c1b Handle component tree failure data 2025-01-22 14:10:27 -08:00
Mateusz Burzyński
dabbfc8bd7
Avoid top level await to fix execution and full module initialization race condition (#9) 2025-01-21 17:52:38 +01:00
Brian Hackett
eee47d9af9
Send simulation prompt commands from client (#8) 2025-01-20 09:49:36 -08:00
Brian Hackett
a5c5d3910a
Update backend APIs for doing simulation based prompting (#7) 2025-01-20 08:03:25 -08:00
Brian Hackett
3f306a26c0
Add menu items UI for about and problem pages (#6) 2025-01-17 14:05:03 -08:00
Brian Hackett
e014056030 Refactor getModelInstance call outside of getStreamTextArguments so we don't need an API key for the selected model if we're using claude 2025-01-16 08:47:07 -08:00
Brian Hackett
f040c9b440 Merge remote-tracking branch 'upstream/main' 2025-01-16 08:02:49 -08:00
lewis liu
41bb909f8d
fix: fallback model name not working (#1095)
Some checks failed
Docker Publish / docker-build-publish (push) Has been cancelled
Update Stable Branch / prepare-release (push) Has been cancelled
Co-authored-by: 刘一奇 <liuyiqi02@corp.netease.com>
2025-01-15 16:36:33 +05:30
Brian Hackett
80ce84f166
Include request body and response header info for fetch calls (#4) 2025-01-14 16:17:47 -08:00
Brian Hackett
6543f33d54
Fixes for cloudflare deployment (#3) 2025-01-14 13:07:23 -08:00
Digitl Alchemyst
85d864f607
Merge pull request #809 from stackblitz-labs/807-transparency-about-development
Some checks failed
Docker Publish / docker-build-publish (push) Has been cancelled
Update Stable Branch / prepare-release (push) Has been cancelled
docs: document how we work
2025-01-12 23:34:48 -06:00
Anirban Kar
fad41973e2
fix: api-key manager cleanup and log error on llm call (#1077)
* fix: api-key manager cleanup and log error on llm call

* log improved
2025-01-13 04:21:29 +05:30
Eduard Ruzga
3a298f1586
fix: cors issues from preview fixed by changing embedder policies (#1056)
Some checks are pending
Docker Publish / docker-build-publish (push) Waiting to run
Update Stable Branch / prepare-release (push) Waiting to run
2025-01-12 21:58:06 +05:30
GaryStimson
6aaff63ca7
fix: bugfix in fetching API Key on base llm provider. (#1063) 2025-01-12 21:54:45 +05:30
Brian Hackett
6bc218340c
Use simulation prompt API (#2) 2025-01-11 13:48:38 -10:00
Oliver Jägle
bc3274c77e
Satisfy the linter 2025-01-07 19:52:50 +01:00
Brian Hackett
c3e1764da3
Merge pull request #1 from replayio/recording-button
Add button to save recording, assorted other UX changes
2025-01-07 05:56:18 -10:00