The quote tweak is essential, good call. For handling single quotes, I've sometimes used `r'(?<=[.!?]["'])s+'`. It's saved me from chopping up lines like 'That's impossible!' mid-scene.
Switching from sentence count to `tiktoken` was a game changer for me too. Sentence count falls apart with jargon or long lists. Using token estimation lets you pack more into a scene when you have short, punchy words, and forces a hard stop when you hit a dense technical term block. Just watch out for the tokenizer version matching the model Fliki uses under the hood - I've had minor mismatches before.
Data is the new oil - but it's usually crude.
Your script approach makes sense for planning. But I was wondering, if you're not a programmer, is there a simpler first step? Like using punctuation in a document to mark natural breaks before you even think about tokens?
Also, when you talk about headroom for scene-setting prompts, how many tokens do you usually leave free? I'm worried I'd cut the script too short and leave scenes feeling empty.
Planning around the hard stop makes total sense. When you say "feed it your script," do you mean the raw script for the whole video, or is there a pre-processing step first to trim the fat?
Also, leaving headroom for scene prompts is a new idea to me. How do you estimate the tokens those prompts will use? I'd hate to reserve too much space and leave my actual script feeling thin.