Why your AI tokens get exhausted.
Ft. A tool to write prompts that consumes lesser tokens.
Imagine this. You’re mid-conversation with an AI. Suddenly, the responses get shorter. Sometimes, worse — the AI starts missing things you said three messages ago.
Congrats! You might’ve hit the token limit.
Most people start a new chat and lose context. We did too. That’s why we built a tool that shows you exactly where your tokens are going — and rewrites your system prompt to use fewer of them.
Set a timer for 90 minutes. We’re starting now.
But first, why do token limits get hit?
Every model has a context window. That’s the amount of information it can hold at once. Meaning input, output, system prompt, conversation history, all compete for the same space.
Here’s why.
1. Conversation history. AI re-reads the entire conversation from the top, every single time you send something. So a productive session hits 50,000+ tokens after just 10-15 exchanges.
2. Pasted documents and code. Simple rule: the bigger and more information-dense your document, the bigger your token count. A 50-page contract is roughly 75,000 tokens. A medium-sized codebase, 250,000. All that even before you ask a single question.
3. Context rot. Models start degrading at 60-70% of their stated window — responses get vaguer, instructions get missed. This is why your AI sounds dumber even when you have memory left.
4. Your system prompt. If it’s 800 tokens of loosely written instructions, that’s 800 tokens spent before you’ve typed a word.
Today, we’re building a tool to solve the last one.
What does the tool do?
It asks you to enter your prompt, tells you the number of tokens it’ll use, and suggests a more token-efficient prompt for you. We’ll build this entire setup using Streamlit, GitHub and Groq. If you’ve read our last edition, you’d recognise that we’d used the same system to build the pitch detector out. But if you’re new here, here’s the rundown of what each tool does.
- Streamlit: Helps us host the webpage
- Groq: This is the AI layer that actually compresses the prompt
- GitHub: This is where our code lives.
- tiktoken: OpenAI’s free library for counting tokens. Runs entirely on your machine. No API key needed.
Step 1: Create a GitHub repo.
1. Go to github.com. Sign in.
2. Click the + icon in the top right corner. Select New repository.
3. Fill in:
Repository name: token-counter
Description: skip it
Set it to Public
Check Add a README file
4. Click Create repository.
Step 2: Create the requirements file.
This tells Streamlit which libraries to install when it deploys your app.
In your repo, click Add file → Create new file.
Name it exactly:
requirements.txt
Paste this as the content:
streamlit
tiktoken
groq
Scroll down and click Commit changes.
Committing is GitHub's way of saving. Think of it as hitting save, but with a record of every change you've ever made.
Leave everything as default and hit Commit changes again.
Step 3: Get your Groq API key.
1. Go to console.groq.com. Sign in or create a free account. No credit card needed, and the free tier doesn’t expire.
Once you’re in, click API Keys.
Hit Create API Key. Name it anything; even “token-counter” works. Copy the key. It starts with gsk_.
Keep this tab open.
Step 4: Create the app file on GitHub.
This is the file that actually runs your tool. It has to be named app.py. Streamlit looks for this exact name when it deploys.
What it does: takes your inputs, counts the tokens using tiktoken, sends your system prompt to Groq, and returns a compressed version with a before/after count.
So, head to your HitHub repo.
1. Click Add file → Create new file.
2. Name it app.py
3. Paste this code exactly.
import streamlit as st
import tiktoken
import os
from groq import Groq
st.title(”Token Counter + Prompt Compressor”)
system_prompt = st.text_area(”System prompt”, height=150)
conversation_history = st.text_area(”Conversation history”, height=100)
new_message = st.text_area(”New message”, height=80)
if st.button(”Count + Compress”):
api_key = st.secrets[”GROQ_API_KEY”]
encoder = tiktoken.get_encoding(”cl100k_base”)
system_tokens = len(encoder.encode(system_prompt))
history_tokens = len(encoder.encode(conversation_history))
message_tokens = len(encoder.encode(new_message))
total_tokens = system_tokens + history_tokens + message_tokens
st.subheader(”Token breakdown”)
st.write(f”System prompt: **{system_tokens} tokens**”)
st.write(f”Conversation history: **{history_tokens} tokens**”)
st.write(f”New message: **{message_tokens} tokens**”)
st.write(f”Total: **{total_tokens} tokens** ({round((total_tokens / 131072) * 100, 1)}% of limit)”)
st.subheader(”Compressed system prompt”)
client = Groq(api_key=api_key)
response = client.chat.completions.create(
model=”llama-3.3-70b-versatile”,
messages=[
{”role”: “system”, “content”: “You are an expert at making AI system prompts shorter without losing meaning. Remove filler, redundancy, and over-explanation. Keep all instructions intact.”},
{”role”: “user”, “content”: f”Compress this system prompt:\n\n{system_prompt}”}
]
)
compressed = response.choices[0].message.content
compressed_tokens = len(encoder.encode(compressed))
saved = system_tokens - compressed_tokens
st.text_area(”Compressed version”, value=compressed, height=150)
st.write(f”Before: **{system_tokens} tokens** → After: **{compressed_tokens} tokens** → Saved: **{saved} tokens**”)
4. Commit the file.
Step 5: Deploy on Streamlit.
Your code is on GitHub, but it’s not running anywhere yet. Streamlit reads your repo and turns it into a live app.
1. Go to share.streamlit.io. Sign in with your GitHub account.
2. Click Create app.
3. You’ll see a pop-up to enter details.
4. Select your token-counter repo under Repository. You should see a dropdown when you click the Repository section.
Fill in:
Branch: main
Main file path: app.py
Click Deploy. You’ll see a screen that says “Your app is in the oven.” Wait 60 seconds.
Step 6: Store your API key safely.
Your app is live. But it doesn’t know your Groq key yet. You’ll add it directly to Streamlit so it never has to live in your code.
1. In the bottom right of your app, click Manage app → Settings → Secrets.
You should see a pop-up.
2. Head to Secrets, delete the contents and paste exactly this. (Replace your groq API key with your actual API key)
GROQ_API_KEY = “your-gsk-key-here”
3. Hit Save.
4. Now go back to your GitHub repo. Open app.py and replace this line:
api_key = st.text_input(”Enter your Groq API key”, type=”password”)
With this:
api_key = st.secrets[”GROQ_API_KEY”]
5. Commit the change.
Streamlit detects it automatically and redeploys — takes about 30 seconds.
Step 7: Test it.
Paste this into the system prompt field — it’s a bloated example we used ourselves:
You are a helpful assistant. Your job is to help users with any questions they have. You should always be polite, professional, and thorough in your responses. Make sure to explain things clearly and provide examples where possible. Never give incorrect information. Always double-check your work before responding.
Leave conversation history and new message empty. Now hit the button. What do you see?
What’s next?
Streamlit, GitHub, and Groq are building blocks. Swap the compression logic for something else — a classifier, a formatter, a scorer — and you have a completely different tool. Same skeleton.
Try something. Break something. Tell us what you built.
Want to plug your brand into our newsletter?
Email us at collab@growthx.club
 | A guest post by
|