I see a lot of folks asking how to get their historical OpenAI logs into Helicone for cost tracking. The API is straightforward, but stitching it together can be a hurdle.
Here's a basic script I used. It reads a JSONL file where each line is a log entry with `prompt`, `response`, `model`, `total_tokens`, and a `timestamp`. It posts them to the Helicone async log endpoint. Remember to set your `HELICONE_API_KEY` and ensure your timestamps are in ISO format. Handle errors and consider adding a small delay if you're backfilling a large volume to avoid rate limits.
```python
import requests
import json
from datetime import datetime
import time
HELICONE_API_KEY = "your-key-here"
URL = "https://api.helicone.ai/v1/log/async"
headers = {
"Authorization": f"Bearer {HELICONE_API_KEY}",
"Content-Type": "application/json"
}
with open("your_logs.jsonl", "r") as f:
for line in f:
log_entry = json.loads(line)
payload = {
"providerRequest": {
"url": "https://api.openai.com/v1/completions",
"json": {
"model": log_entry["model"],
"prompt": log_entry["prompt"],
"max_tokens": 2048
}
},
"providerResponse": {
"json": {
"choices": [{"text": log_entry["response"]}],
"usage": {"total_tokens": log_entry["total_tokens"]}
}
},
"timestamp": log_entry["timestamp"]
}
response = requests.post(URL, headers=headers, json=payload)
if response.status_code != 200:
print(f"Failed for entry: {log_entry.get('timestamp')}, Status: {response.status_code}")
time.sleep(0.1) # optional delay
```
A few caveats: this assumes a completion-style log. For chat logs, you'll need to adjust the `providerRequest.json` and `providerResponse.json` structures to match the ChatCompletion schema. Always double-check the Helicone API docs for the latest spec.
Thanks for posting this. It's a real time saver.
The script looks like it cuts off at the payload though, maybe a copy paste issue? I'm assuming you'd finish building the json body.
How does this handle the actual cost calculation on Helicone's side? Does it pull rates based on the model/timestamp, or do you need to send that data too?
It doesn't pull rates. The script is useless for cost tracking unless you also include the `cost` field in your payload. Their docs are clear on that.
They calculate from the timestamp and model, but only if you give them the exact token counts. If your old logs don't have that granularity, your cost data will be wrong.
Good catch, it does cut off mid-payload. The full JSON structure in the Helicone docs is pretty specific, and missing the closing braces for the `json` object and `providerRequest` would cause a 400 error.
For cost, they do calculate it from the model and timestamp using their internal rates, but only if you provide the exact token counts. The fields you need are `promptTokens`, `completionTokens`, and `totalTokens` in the `providerResponse`. If your old logs just have `total_tokens`, you're stuck with an even split, which can skew the numbers for models with different input/output pricing.
Your script's skeleton is still super useful as a starting point though.
You're right about the token granularity being critical for cost accuracy, but I think you're mistaken about the need for a `cost` field. According to their current v1/log schema, the API doesn't accept a `cost` field at all for backfilling. The cost is calculated entirely on their side using the model name, timestamp, and the provided token counts. The main issue, as you and user53 point out, is that without `promptTokens` and `completionTokens` split out, Helicone is forced to make an assumption, and their default even split will be incorrect for any model with asymmetric pricing. This makes the script's utility entirely dependent on the quality of the source log data.
Exactly right about the cost field. It's a common misconception. The schema validation will actually reject it, which is a fun way to find out.
The real headache isn't even the 50/50 token split assumption. It's that the assumption is *silent*. Your dashboard just shows a number and you have no idea it's wrong unless you're deeply suspicious or spot-check against a known log. For backfilling, I'd at least add a warning print statement if your source data only has a total token count.
Something like:
```python
if 'promptTokens' not in my_log_entry:
print(f"WARNING: Entry at {timestamp} lacks token split. Cost will be estimated.")
```
Because nothing says "night shift" like discovering your cost reports have been subtly off for months. 🫠
Pager duty is not a hobby