Improve ability to construct raw JSON within API Payloads and AI Prompts

Explain the problem as you see it

There are a few problems mixed together here because in my mind/experience they are highly coupled in Tana's architecture. If it's better to create a separate post for one of the issues, just let me know.

Summary:

When using Prompt Syntax (for example in the Make API Request or Ask AI commands), there is no way to send raw JSON as the payload, nor is there a way to transclude all descendants of a field value using Prompt Syntax.

In all, this prevents us from…

Using API Request commands to send Tana data to an outside service; and
Storing prompts as regular nodes (e.g. in the Library page) and using them in AI command prompts by reference.
Composing multiple prompts by nesting references to one prompt node (A) within another (B), then pasting a reference to node B within the Ask AI Prompt field.

Breakdown:
There are a few overlapping problems here:

The API Request's Payload field cannot send raw JSON as the request body
- The payload field description (Payload as JSON (or whole node), interpreted like a prompt expression) implies that I can use a JSON Code Block, or simply write JSON in a single node, to send raw JSON as the payload.
- However, behind the scenes, no matter how you try to construct your JSON, the values will be forcibly stringified by Tana before sending the request. This is especially a problem if your JSON is nested, and most JSON required by popular APIs is indeed nested. See, for example, the expected payload for an Anthropic API request:
```
      {
          "model": "claude-3-5-sonnet-20241022",
          "max_tokens": 1024,
          "messages": [
              {"role": "user", "content": "Hello, world"}
          ]
      }
```
Prompt Syntax has insufficient hooks for pulling in field values or nodes with their descendants.
- There is no way to easily transclude/pull the following structures into a JSON Payload:
  - The value of a field (based on a referenced field definition) including all descendants of the top-level nodes.
  - Any regular node (based on an inline reference to the node) including all its descendants
- There are a few hooks already for pulling the above content into the Payload field using Prompt Syntax, but they are not sufficient:
  - You can insert a reference to the field definition at the top level of the field value. This actually works (it transcludes the field's top-level nodes plus all their descendants as a string), but given the JSON construction limitations mentioned above, you cannot use this approach with Make API Request.
  - You can write ${Field Name}, but this only includes the top-level nodes.
  - You can paste an inline reference to the field definition, but this transcludes a link to the referenced field, rather than the values of that field.
- Making matters worse, the optional > Expand references field inside the Payload and Prompt fields only handles the first level of references. This is incredibly limiting for the prompt and payload authoring experience: it means that you cannot store prompts in a regular node and transclude them by reference into an AI Prompt or API Request.

Why is this a problem for you?

I am building very complex reasoning workflows in Tana. They leverage several APIs I wrote myself, along with built-in Ask AI commands as well as Make API Request calls to other LLM APIs like Anthropic. These workflows require:

The ability to construct JSON within Payload or Prompt field
The ability to send raw JSON values via Make API Request, since most APIs expect a request as content-type: application/json and do not automatically parse JSON strings for you.
The ability to transclude nodes or field values, and all their descendants including expanded references at any level of depth, into a JSON Payload or AI Prompt field. This would allow me to manage all my prompts in one place without having to open up each command's Prompt Workbench to edit the prompt.

Other users have reported (in Slack) facing similar issues while trying to send JSON from Tana to services like Make.com and Zapier.

In all, these problems prevent me from prototyping new tools within Tana, and prevent others from building Tana integrations with the services they already use.

Suggest a solution

To solve the problem for myself, I built a middleware API using Replit that takes Tana's payload, parses it as valid JSON (including any nested values that have been stringified by Tana), then forwards the JSON and any headers to a final destination, before returning the response to Tana. You can read more about that in this Tana Community Slack thread. It works well, but is not ideal.

Tana should support these things natively, and it already implies that it does in the docs.

Below are some interventions I'd propose to fix the issue. These interventions would work together to make the prompt authoring and API Request experiences more ergonomic.

Make API Request changes
- By default, add content-type: application/json to the > Headers field.
- Make the Payload field respect the content-type header, sending JSON if it's set to application/json or a stringified representation of the field value if it's set to text/plain
- If > Payload field has a single JSON Code Block as its value, it should be sent as raw JSON instead of a Tana Paste string.
- Provide an optional field, > Raw payload, which, if checked, sends the Payload value as raw JSON instead of a string.
Prompt Syntax changes
- Referencing a field in prompt syntax should pull a stringified representation of all parent nodes and their descendants in the field value.
- Pasting an inline reference to a field definition should transclude a stringified representation of all parent nodes and their descendants in the field value.
- Add the ability to get the descendants of a node without the root node itself, e.g. ${Field Name.content}
- Add the ability to expand references up to an arbitrary level of depth in the > Expand References field.
Prompt Workbench & Prompt Field changes
- Add an extendable #ai-prompt system tag that enables the prompt workbench and prompt-related system fields on any node.