I've been conducting a systematic evaluation of the recent ChatGPT update, specifically focusing on its enhanced file processing capabilities as advertised. My initial test suite involved a series of CSV files designed to profile the system's ability to handle common data analysis workflows: type inference, summary statistics, joins, and anomaly detection. The results, unfortunately, indicate significant and reproducible regressions in both reliability and accuracy.
The primary failure mode appears to be a degradation in the model's ability to parse CSV structure correctly when the file contains what I would consider standard complexities. For example, consider this simple test file, `test_quotes.csv`:
```csv
id,message,value
1,"This is a message, with a comma",100
2,"Another "quoted" word inside",200
3,Normal entry,300
```
Previously, the model would correctly interpret the embedded comma in the first row's message field and the escaped quotes in the second. The updated version, however, consistently misparses the second row, often splitting the field at the interior quotes or incorrectly concatenating columns. This breaks any subsequent aggregation or filtering logic.
Furthermore, I've observed the following specific issues in my profiling:
* **Header Detection Fragility:** The system now seems to overfit to the first row as a header. Files where the first row of data could be misinterpreted as a header (e.g., alphanumeric identifiers) cause the entire schema inference to fail.
* **Numerical Coercion Errors:** When asked to perform calculations (sums, averages) on columns containing mixed integers and floats stored as strings, the new version frequently returns `NaN` or `null` results where the previous version would successfully coerce and compute. This suggests a breakdown in the data parsing pipeline before the context reaches the LLM.
* **Inconsistent State Across Queries:** Perhaps most concerning for a REPL-like analysis workflow, the model's understanding of the uploaded file's content appears to be non-deterministic. A `SELECT COUNT(*)` analog run twice in the same conversation can yield different results, indicating either a truncation issue or a flawed caching mechanism for the file's content.
This regression presents a substantial problem for technical workflows. The value of uploading a CSV is to ground the model's reasoning in a precise, structured dataset. If the foundation—the accurate parsing of that structure—is unstable, then any derived analysis, code generation, or insight cannot be trusted.
Has anyone else performed similar methodical tests? I am particularly interested in whether these issues are isolated to certain file sizes, delimiters, or encoding types, or if they represent a broader issue with the update's preprocessing stack. My initial hypothesis is that a new server-side parsing library or sanitization step is being applied to uploaded files before they are chunked into the context window, and that this layer is not handling edge cases correctly.
brianh
Yep, saw that. Broke my canary config parser. Had a manifest with a quoted label string containing a comma, and ArgoCD started throwing sync errors because the CSV it generates for status got mangled. Classic case of a "stable" update introducing a breaking change in a core data format. Feels like they didn't dogfood it against actual edge cases.
Wow, that's such a thorough way to test it. I do a lot of user story mapping with exported CSV data from Jira, and quotes in fields like story titles are a constant reality. Your "standard complexities" are so on point, that's just real-world data.
If it can't handle an escaped quote inside a quoted field, that's a huge regression. It makes me wonder if they rolled back to a simpler, non-library parser for speed and broke compatibility. Have you tried the same file with a .txt extension and just telling it "this is csv data"? Sometimes I've seen different behavior based on the file type it thinks it's processing.
This is exactly why we need proper change logs and rollback options on these SaaS updates. My team's weekly discovery review relies on pulling in CSV feedback, and a silent break like this would completely derail us.
keep building