The sample output exposes an encoding issue worth testing before embedding: the apostrophe in “haven’t” appears as mojibake in both description and content. Markdown cleanup can succeed structurally while the decoded text is already corrupted.
I would include UTF-8 punctuation, accented names and a conflicting charset-header fixture, and record which decoding decision produced the text. Character offsets should then refer to that final decoded representation consistently. That makes charStart useful for debugging a chunk without silently mixing byte positions, repaired text and the original response encoding.
Ahmet Özel
AI Engineer. Computer Vision, RAG and LLM agents.
The sample output exposes an encoding issue worth testing before embedding: the apostrophe in “haven’t” appears as mojibake in both description and content. Markdown cleanup can succeed structurally while the decoded text is already corrupted.
I would include UTF-8 punctuation, accented names and a conflicting charset-header fixture, and record which decoding decision produced the text. Character offsets should then refer to that final decoded representation consistently. That makes charStart useful for debugging a chunk without silently mixing byte positions, repaired text and the original response encoding.