Pensabot: how I manage my notes with Telegram and AI

In general, I like taking notes on paper for day-to-day things and using Obsidian to manage the projects I am working on. The problem is that I find it difficult to keep everything organised or capture an idea.

That is why, some time ago, I experimented with the idea of having a Telegram bot that would let me send it ideas, voice notes, or links and have its own memory, so I would not have to worry about managing the content.

This way, I talk to an AI agent that decides whether to store information in its memory or reply with information I previously sent it.

From a proof of concept to something useful

This is my new side project: Pensabot (GitHub repository). It uses OpenAI’s GPT-5.6 Luna as its foundation, which performs very well at a relatively low cost. So far, I have spent no more than a few cents for the way I use the bot.

The main challenge is information retrieval. I started this project a few months ago, while I was experimenting with and learning how to manage AI agents, when harnesses were not as advanced as they are now. It is a field that is evolving very quickly.

Pensabot was almost a “one-shot” implementation of a specification that I refined with AI. By now, it is clear to me that one-shots are useful for quickly creating a proof of concept, experimenting, and learning from something tangible, but they then require a lot of refinement if you want to put the result into production.

The good thing is that code generation is so inexpensive that you can experiment as much as you need, at a low cost and in very little time, until you have something you are happy with.

Over the past few days, I have felt like returning to this project and rethinking it from scratch with everything I have learned. I started with a much smaller version in which you can send a message through Telegram and receive a reply.

A memory with two tools

For now, the memory system consists of a messages table that stores the conversation with the user and allows the agent to include the user’s most recent conversation in its context, together with a separate memory table in which the agent creates memories about the user as the conversation develops.

To do this, the agent has two tools that it can decide to use:

  • Add information to memory. The agent decides which information from a message it wants to store in memory. It can store multiple ideas separately.
  • Search the memory. The agent can search its memory for information related to the user’s query.

The idea is for the agent to be completely autonomous and decide which tool to use and how to use it.

Diagram option: an idea that is stored and retrieved later.

sequenceDiagram
    actor user as User
    participant telegram as Telegram
    participant agent as Agent
    participant memory as Memory

    user->>telegram: Remind me to write about PostgreSQL
    telegram->>agent: New message
    agent->>memory: Store "Idea: post about PostgreSQL"
    memory-->>agent: Information stored
    agent-->>user: I'll remember that

    user->>telegram: What did I want to write about?
    telegram->>agent: New query
    agent->>memory: Search for post ideas
    memory-->>agent: Idea: post about PostgreSQL
    agent-->>user: You wanted to write about PostgreSQL

For this first version of search, I preferred to start with something simple that works, and then try more complex strategies, such as semantic search or hybrid search.

Based on the conversation, the agent decides whether it needs to search the memory for information. If it does, it generates a query that is passed to a tool. It is important to highlight that the agent autonomously decides which words to search for based on the user’s intent.

The function splits and preprocesses the query text (under the hood, it uses PostgreSQL’s to_tsvector('simple', query) function) and ultimately produces a ranking of the user’s memories based on the number of matches.

I am surprised by how well it works despite being so simple. Over the past few days, I have been using it to jot down ideas for posts or things I want to try. I asked it to find some specific ones and sort them, and it did a good job. It makes me wonder how it will work as the memory fills up.

What I am enjoying about this side project is that it is helping me understand how an agent works, how tools are integrated, and how difficult it is to manage memory and context well.

In future posts, I will explain the changes I am working on in detail. You can clone the repository and set it up on your own computer or homelab.