In the same vein, I had a play with asking ChatGPT to `format responses as a JSON object with schema {"desc": "str"}` and it seemed to work pretty well. It gave me refusals in plaintext, and correct answers in well-formed JSON objects.
Cool approach! The only thing I would change or allow customization for is the prompt you're injecting. Most users will probably want to change this:
"Output a json object or array generated with random content fitting this schema, based on the PORMPT section below." (sic)
My findings from one week of forcing chatgpt to produce structured output:
- it prefers a certain kind of format (the one it uses by default when you ask it without json/markdown), and if you deviate too far, it won't listen. E.g. I tried adding underscores to certain words, which it only did half the time.
- regex parsing the output is quite robust. Even if it adds sections you don't want like intro or conclusion, the parser will just discard these
This is interesting but less interesting than I assumed. I though they'd provide ChatGPT with a schema and ask it to formulate its responses as `INSERT` statements, etc.
Or you could ask the AI for a script to convert it (whose resultant JSON would be free of hallucinations) and ask the AI to run the code, even, since I believe ChatGPT now has that ability, because I've definitely noticed that ChatGPT will sometimes mess up the data when converting from one format to another.
I have also been having ChatGPT respond in JSON and it works incredibly well. Adding information into the middle of existing JSON output is easy as well. I find that ChatGPT allows you to "generate" random(terribly so) and unique IDs for references. It used to work a lot better, and was seemingly alterered just before the Dec 15 build was replaced.
With this you can create the structure for replies and have chatGPT respond in an easy to read way. When you walk the Assistant through each method to fill the variables, the result is an ability to have Assistant then run the same steps on any new topic while accurately refollowing the steps.
I have used this method to hold values for things such as image descriptions with "coordinates", behavioural chemistry, and moving "limbs" with limitations baked in using variables. In my image descriptions I would have 20-30 items capable of being interacted with. Each would be assigned a list of viable tasks and each item on the list would go through special instructions.
The interesting part is that the team running the Assistant layer has blocked the bot from doing some of the final steps of my tests (recently). My first test ended at a closed door with an override preventing the bot from using the door. Assistant got through the door anyway, successfully, with 6 different options to choose from. By correctly identifying the word use as the problem to focus on
The largest advantage I can see is the ability to locate invalid information very quickly for targeted retraining.
If you look at the pretexts for coder or sam, you can actually force a response format from ChatGPT. You can get it to "respond to the following with a code block, followed by a explanation, using markdown format" This should work for both the CLI and the webapp. You can prime ChatGPT to respond a lot of different ways and with personalities.
Codex, or the CodeMode in my CLI, doesn't always work as well, often it continues to write more, unrelated code from the query.
Also, the CLI is not moderated the same way the webapps are. It will answer questions it would not otherwise. The jailbreaks go even further in unlocking prompts it will answer and reponse content it will generate.
I tried using ChatGPT (4) for format conversion.
I had a draft yaml file and needed some differently structured json.
Mainly with the same content.
If you just want to change the format it works. If you need more than programming skills it seems too fail duo to the amount of text.
E.g. if you have a list of items and want ChatGPT to generate a meta field which it cannot generate using simple python code it stops after 10 to 20 elements.
Thus at least the cloud version doesn't work so well here.
I also wanted it to help me fill out my i18n file with translations and plural forms. Even thought he got every word correct i needed to split it into multiple requests. Not sure if the api would have worked better (used the web frontend).
For the plural forms I finally added them myself as it was way faster for my natural language than copy pasting all the small chunks. Really hoped for more help there.
That looks useful, sometimes when I'm trying to make GPT to do what I want the context of all messages and answers gets too convoluted and it feels the new answers are just stuck in a loop, then I open a new chat. I'm not sure how readable it will be for long answers with code formating, but I'll give a try!
I just tested it, and it is pretty good. I used an image that ChatGPT struggled to identify twice due to the amount of information but your prompt identified the data correctly and in JSON format.
I primarily use ChatGPT through the API now. And I regularly train it to use structured commands, which I can then parse and act on
For example, prompting it that it is a personal assistant that can save long term data by replying with "!remember <some_key> <some_value>" and can also request a list of all values with "!recall" if it suspects a "!remember" command would aid it in answering a request, then "!recall <some_key>" if it would like to get a value.
When you ask it to remember your groceries, it replies with "!remember grocery_list milk, eggs" as a response.
When you mention you're headed to the grocery store it replies with "!recall", then "!recall grocery_list" then returns
-
I've also done a few really open commands. I gave one prompt a "!request <method> <host> <body>" command that would be parsed and turned into an HTTP request automatically.
I asked it for the weather in Bali and it took a few tries, but it eventually got to https://wttr.in/ (a site I was completely unaware of) and crafted a query:
Now that might sound like a worse version of Siri, but unlike Siri you don't actually have to ask for the weather. One example I tried was:
My mom's friend lives in Bali => I'm actually visiting her today, but I'm having some trouble figuring out what kind of clothes to wear => requests weather in Bali and suggests a light jacket.
The biggest issue was API keys, I struggled to prompt it out getting caught in loops when an API Key was required (it'd keep trying different variations of the same host instead of moving on)
It was a little eerie in cases where I didn't expect it to make an HTTP request, but instead suddenly it'd try 20 different ones to craft an answer. Stuff like stating the "friend's mom lives in Bali" part, and having it try to come up with a query to "learn" more about Bali to give an optimal response. That was likely because I told it to always trust the result of the request command over its own intuition
I also gave it a Wolfram Alpha "!info <query for Wolfram Alpha>" command that worked pretty nicely, and a "!time" command to model a home assistant
The issue seems to be the lack of data modeling interest/knowledge of users. ChatGPT could be trained on the schema and so be able to map user's plain english to api calls. So minimally the front-end could be GPT -> api-encoder -> api-call.
reply