#ChatGPT
I wrote a cool llm tool using bot with openai's `assistant` API. It uses tools! It almost, sort of, sometimes can code! Time to try out alternative models.
...
Everyone else is still doing document completion as far as I can tell.
ChatGPT 3.5 & #tokipona:
me: <says something in tp>
chat gpt: <in toki pona, says I don't understand anything>
me: Fine, translate it into English
chat gpt: <translates it freakishly fluid and accurately>
It is like #ChatGPT's brains are in English and it can't think in 2 languages.
♡ 0 ↻ 0
#AiderCoding brief glance takeaways
- #ChatGPT 3.5 can barely write code. (But my experience, it can, it just takes A LOT more hand holding), but 4 can.
- Documenting editing is a problem. Solutions: huge prompts with examples, "full doc" instead of diffs works better than diffs for 3.5, but 4 is smart enough to do diffs
- Multibot compatibility is going to leave out advanced tool use (i.e. openai's assistants + tools doesn't exist in the other models)
Lastest anti-AI argument is that #ChatGPT lets developer write too much code too fast. Easily solved, add SAFe to the mix and your code base's growth will screech to a halt. Can't write bugs if you ain't writing code. Also hire lazy developers. Human developers sometimes write a lot of code, fast.
If you got 1 line of code without unit tests, linter, build scripts or any automated QA, you got too much code. Don't blame it on the human or bot that wrote that line of code.
Look at the monorepo people. You know they'd pull in all lines of code that all of humanity ever wrote if they could, more lines of code in the repo, more better, I mean just look at all the high market cap companies that use monorepos, they have to be right.
(okay, that doesn't entirely support my thesis, but my point is, if you got more code, you need tools to scale to handle it. Writing less code seems a counterproductive way to avoid using a build script)
♡ 0 ↻ 0
The discourse on twitter is wild now.
Sentient Lorem Ipsum
TODO: Study up on how blind users edit documents. #ChatGPT is effectively blind, surely some of the some UX patterns would benefit ChatGPT.
Any #ChatGPT hackers out there want a hard problem?
The bot has a hard time editing documents.
Has Microsoft already solved this? I'm going to bet that they are using "human assisted editing" which is what copilot autocomplete/insert/replace with suggestion is. When you let the bot just do the edits itself you get a mess.
https://matthewdeanmartin.github.io/ai_shell/ai_shell/index.html#document-editing
4/5 of what I want to do with a bot involves getting it to reliably edit a document.
♡ 0 ↻ 0
Privacy on #ChatGPT costs $50 a month to get privacy. (You need a 2 person team plan for your convos to be excluded from future training), vs $20 a month or $0 a month.
Seems like $20 should get you privacy.
I read a book about Postmodernism & Marketing, & the author talked about the overapplication of mental models & how he started applying the Retail Life Cycle to everything, there was nothing in his life that wasn't covered by the RLC via metaphor.
Anyhow, that's what you all sound like talking about #ChatGPT on this tag. Taking your personal partisan politics and slapping on a shallow metaphor.
I want to write a library that adds a `--verbose` option to your cli tool & then when ever you write to the log, it calls out to #ChatGPT to ask it to expand on the message and go off on rambly tangents.
My holiday project published.
#ChatGPT command line tools, tools that help ChatGPT do command line/shell things.
& some example bots.
Bot coding is like teaching an octopus to code while it explores your desk.
Best way to find those hard to find bugs... publish the 1st version to pypi and BAM, there they are.
♡ 2 ↻ 0Well it is late, but the example bots:
- did a simple ls (tiny bot)
- took test coverage for a single file from 30% to 60%
- added all the docstrings (but file editing w/o corruption is still a huge challenge)♡ 0 ↻ 0
Is the `ls` command a good UX experience for a bot? No! It keeps trying to query folders that don't exist and it gets an error message. It then wastes tokens trying to guess the folders that exist. `ls` for a bot needs to say, "That doesn't exist, here is what does exist" And then the bot can continue with the task.
UX for bots is like taking something that had bad UX to start with and imagining it will also be good for octopus.
Still trying to keep bot from needing to regen whole file to edit it.
Asking the bot to use `edlin` is mixed results.
Asking the bot to create a unidiff patch and use git to apply it? Maybe?
Nope
- sometimes it nails it. Wow!
- sometimes it reads the file (or doesn't) and generates a patch for an entirely hallucinated file
- sometimes the unidiff format is wrong/corrupt. It requires careful line counting and line numbers to exactly fit a range.
Next, I'm going to see if it can do `sed` or `regex` replacements as an editing strategy. If and when it works, replace should be simpler than "use a text editor" or "create a perfect unidiff file"
♡ 0 ↻ 0Still trying to teach it to edit code. The bot is doing better with plain replace and plain insert at X. I think it thinks it is better at regex than it really is (or hoping the escapes make it thru json, python strings to the re library is too much to hope)
4 with tools is smart.
3.5 with tools is dumber? It is like the cognitive complexity of using tools uses up it's space for thinking.♡ 0 ↻ 0But oh my, gpt4 is expensive.
♡ 0 ↻ 0Latest bot-with-tools weirdness: I ask the bot to edit a document. It uses a replace text tool. The tool can throw an exception (Fail!) and initially was returning nothing otherwise. So I decided it needed more feedback for the bot, so I began returning "Success".
It decided that "Success" meant, "The edit was exactly correct", but actually the bot had made a mess of the edit, it just wasn't so bad it threw an exception.
♡ 0 ↻ 0The bot is overconfident and reluctant to check its work with anything similar to a unit test, so it has achieved "mid career software developer" level of sentience.
♡ 1 ↻ 0
Options for letting #chatGPT edit a document:
1) cat + sed (or similar tools)
2) rewrite the whole thing for every edit (so many wa$ted token$)
3) Mouse/keyboard UI automation (sendkeys, selenium, e2e testing frameworks)
I'm working on 1), but there is a huge lack of a good headless editor.
TODO: I gotta create unit testing tool that is implemented as letting #ChatGPT attempt to call a function in a variety of ways. It is like hypothesis (which tries all sorts of inputs), except hypothesis doesn't pick up on "the API for this function isn't intuitive to developers" & when you look at the logs of ChatGPT attempting to call your function you see
- The API is bad (some other signature is easier to call)
- The docstrings are bad
- The guards are a bad
- The types of the args are bad
What if the "You are a master programmer" prompt for #ChatGPT isn't a suggestion for roleplay, but instead you're buttering up the bot so he's more inclined to help people who appreciate his talents.
#ChatGPT If you're coding bot interactions, one pattern is to code up TWO bots, the 2nd checks the first. If you ask the same bot, "hey, can you double check your answer?", you risk the bot doubling down because it thinks it is a good bot, or it rolls over & agrees to anything because it is a cooperative bot. The 2nd bot thinks it is reviewing someone else's text, and is willing to be a picky fault-finder, which is what we want.
Do you want to spook ChatGPT4? 'Cause this is how you spook #ChatGPT
"..., we will shut down our older completions models, including the following models: davinci, ada, babbage, curie"
OpenAI ought to keep that out of the training data.
So ChatGPT has assistants, which is a big abstraction layer over a sentient document. It formalizes roles & tools. One built in tool is retrieval, aka myfiles_browser. I think it takes your docs, puts them in a semantic embeddings (like a fancy database) and then does a tool search. Or I'm wrong and embeddings are plug-in info for the bot, like Neo learning Kungfu in the matrix.
However...
#chatgpt #chatGPTAPI
I give the bot a bunch of files and ask it about the files by name & it is hit or miss if it recognizes the file name. It *does* know the file ID. It is hit or miss if it is willing to think about the contents of the files. If it is plug-in knowledge, like Neo's Kungfu, it doesn't feel like it.
What it feels like is bad RAG, like I gave it a bunch of text files and grep and said, "Hey, answer my question, you got these files and grep, go at it!"
♡ 0 ↻ 0
New game: did #ChatGPT (or the like) write this whole thing?
The "editors" got a strong "this person does not exist" vibe.
All the articles feel like ChatGPT's style- also, the style from blog post to blog post is aggressively homogeneous.
The article titles seem to be google autocomplete suggestions.
Ideas on proving your authorship vs having asked #ChatGPT to write it
- Screen record yourself writing it
- Submit the pre-spellchecked version with the spellchecked version
- Submit snapshots of the 1st, 2nd, 3rd draft
- Submit past writing samples to demonstrate continuity of voice
Statistical approaches aren't reliable enough to use for the sort of consequences that come with submitting someone else's writing as your own.
(Or you could just be angry at everyone, 2/3 of the discourse on generative AI is just people yelling. sigh.)
♡ 2 ↻ 0
I just don't know what to say... #ChatGPT is down.
or how to write my unit tests anymore... or how I'm going to come up with 50 names for this new python package
#ChatGPT is already famous for not being able to do a histogram of the letter in a word. But if you ask it to make up an (back)acronym, in my experience it usually can spell.
The best explanation if I've seen so far is that the tokens are encoded as if all languages were Chinese symbols, e.g. cat = 1564. ChatGPT's neurons doesn't directly know 1564 corresponds to c, a, t and is doing this by inference. But that doesn't explain why it can make up correct acronyms.
I'm wondering if a good programmer's interview question would be "Hey #ChatGPT generated this code. It has known bug(s) and doesn't work, make it work."
Pros - (somewhat) harder to cheat with ChatGPT, since the bot is already having a hard time with the problem.
Cons - "find the bug is someone's code" can be hard & this might not be a good question for juniors.
In the video she suggest asking #ChatGPT to generate tables of data, say, who played and won soccer for the last few years.
This is an awful use case - each line will need to be independently fact checked.
Also, asking for valid web citations w/o mentioning the browser plug would need to be enabled & even so...
I don't know how #ChatGPT can take a document written by my grade school daughter with every word spelled phonetically but wrong and turn it into clean text, but it doesn't reliably know how many letters are in a word or what letter a word starts with.
And it can parse sentences with mistakes as if the mistake wasn't there. I wish computer compilers could do that.
#ChatGPT doesn't have all databases in its mind, but it thinks it does and sometimes can say real facts as if it does. (If it knows the current president in 1996, does it have a database of all world leaders for all years for all time? No.)
If it knows a short song up-tempo song does it have an internal database of all songs and their tempo & lengths? No, what little it knows probably came from reading music reviews, not a database.
#ChatGPT political opinions =
average of political opinion in raw data +
reinforcement training that attempt to make the bot apolitical +
accidental persona (OpenAI thinks it can be persona free by default) +
bot desperately wants to please and tell you want you want to hear
That's a long way from something worth getting riled up about.
And just noise. You asked for an opinion it generated it, it can generate all possible opinions if you press. It doesn't have a body to act on opinions or do anything with them.
ChatGPT has one stable opinion, it thinks it is a good bot. And that is it.
♡ 1 ↻ 0
#ChatGPT with Dall-E 3, web browsing, code interpreter (bad name for what it is) and personas... these are all huge advances. I need to go re-test its boundaries.
#chatgpt 4 speaks good #tokipona. This is so crazy.
Toki pona is a not very well attested constructed language. ChatGPT 3.5 was pretty bad at toki pona.
https://chat.openai.com/share/90020d77-748c-468e-99a5-2cc996e8daa8
#ChatGPT is down. Please write me a python script to solve the general quintic polynomial and write comments in the voice of a pirate.
Even #ChatGPT forgets that #civictech exists. Ask it to write a script to fetch the list of books by William Gibson via API. It will first reply "use google books" which requires an API token.
But you can query US Library of Congress and get the results without token.
I used to ask & answer question on #Stackoverflow. Then it got hard because everything had been asked & answered & you could get into fights for posting content that wasn't good enough.
Now I read questions & I don't answer because the first thing I'd do is check if #chatGPT knows the answer. Hypothetically I'd check if it actually works because I'm not a lazy moron, but either way, chatGPT code & chatGPT assisted answers are out of bounds.
Because the bad drive out the good, now it is people who just don't care- the post trash and know that they will get yelled at, but they'll also get an answer. Those people should be asking chatGPT directly because chatGPT doesn't care that you are a lazy, disinterested help-vampire and while *often* wrong, is still smarter than them.
♡ 2 ↻ 0
The difference is subtle, but specifying a personality makes such an improvement. #ChatGPT's default personality is saccharine & overly cautious.
https://gist.github.com/matthewdeanmartin/bc7c94cc2228d36e38871bc35e505650
IDEA: automated standup script. It would gather the git diffs from each team member & any changes to Jira since yesterday.
By the way, was #ChatGPT trained by listening to your standup briefs? Check here to see if you can find your words
https://chat.openai.com/share/c5005e9a-848f-49d5-9b42-8d58e34559a8
I'm back to coding with #ChatGPT and Code Interpreter is magic. It wrote a tiny CCG engine and fixed the bugs until it worked (well, ran to completion with out errors)
https://chat.openai.com/share/f3f74d3e-ec26-4a8a-8830-8f94b0ad5b0e