Leading AI

The current token budgets are untenable.

Or you could say the rate at which the new models use tokens is untenable.

With the standard ChatGPT business account, I can work for about an hour until my 5-hour budget is totally blown. And that’s not even using the frontier model Astra. That’s with the (now old) Terra at medium context. With Astra, its more like 20 minutes.

Work for an hour, then wait four more until my 5-hour budget resets. Then do it again.

So that’s two hours of work. Three if I spread it out across another 5-hour window.

You can’t take off working like this.

What are your options?

• More tokens. You can always throw money at the problem.
• Get better at which model to use for what task. You should do this anyway. It helps, but it only takes you so far.
• Break jobs into smaller tasks. Context costs tokens. Less context, less tokens.
• Avoid duplication. Models, especially when you have multiple agents running. If you don’t manage what each is doing, they’re probably each rereading all of the same context.
• Build efficient, AI-readable memory storage. If you cut down what the model needs to read to understand, you use less tokens. This goes hand-in-hand with the one above.
• Build deterministic software tools to perform much of the work. Use the AI to generate reusable code, rather than asking it to continually use a model to do the same work over and over. This is software development 101. Write code.
• Do more yourself. Duh.
• Ask better questions. Give better instructions.

When you look at this list, it looks a lot like what a good manager does with his/her team.

Once again it comes down to being a good leader.

In some ways (not all ways), leading AI isn’t much different from leading people.