Imagine curling a wikipedia article. Ordinary code can strip out the HTML tags...
@jamiemccarthy Imagine curling a wikipedia article. Ordinary code can strip out the HTML tags (e.g. html2text) for free. It will still have header and footer crap, which could be 1/3 to 2/3 of the body. So that triples your GPT-4 API costs. Compared to 4, ChatGPT3.5 is free & can extract the important bits before we give it to GPT-4 to ask a more difficult question.
Self-replies
@jamiemccarthy The technique generally worked, except the bot can't hit a token goal without multiple retries and the AI generated JSON keeps getting truncated. (Already there are 3+ pypi packages for repairing truncated json and all of them mention openai as a use case)