← Blog

River, Compass and How My Bots Learn

Shopify and Wealthsimple each run an agent in Slack. They differ in who gets to see an agent’s work before it spreads. Among my own bots, a lesson moved on only when it came up while I was in the chat.

On Thursday, October 29, I went to the DET Toronto Meetup, run by Data Engineer Things at Shopify’s Toronto office. Shopify’s speakers talked about River, their internal coding agent. The speakers from Wealthsimple mentioned their own agent, Compass, in passing.

River lives in Slack. People ask it for help by typing @river in a public channel, and it reads code, runs tests and opens pull requests (proposed code changes). Compass is what Wealthsimple calls its AI teammate. It lives in Slack too, and it also picks up Jira tickets and GitHub comments. It writes code, answers questions about the company’s data and opens pull requests.

At first I thought the repositories set the two apart. Shopify keeps its code in one repository, and Compass works across many, so a repository looked like a wall around what an agent can reach. I run a small team of bots on Grok Bot, and each bot has its own repository, so my setup looked more like Compass than River. After the meetup I read what both companies have written about their agents, and the repositories explained less than I expected.

How far each agent could reach

River works from the root of Shopify’s one repository, which Shopify calls World. Shopify bet on that single repository in early 2024, about two years before River launched, and its post about River’s internals says this bet “made it all possible.” From the start River could see everything in World, because there was only one place to look.

Compass started small. Wealthsimple’s post about it says it “worked in just a handful of repositories” at first, and engineers kept asking the same question: “When will this work in my repo?” The team tested it, widened its reach, and within months Compass worked across more than a hundred repositories.

Wealthsimple has a large shared repository too. Dennis Brotzky joined the company when it bought his startup, Fey, and in a personal post about his first year he wrote that all of Wealthsimple’s client-facing code lives in a single repository, the largest in the company.

So the two companies do not split neatly into one repository and many. Still, River could reach everything in World on its first day, while Compass grew into its reach over months as teams asked for it. The repositories decided how much code each agent could reach and how fast it spread through its company. To compare the agents themselves, I had to set the repositories aside.

The parts the two agents share

Side by side, the two agents look almost the same. People ask both of them for help from Slack, in the middle of a conversation they are already having. Both pick up new abilities from skills that the teams closest to the work write for themselves. Tobi Lütke, Shopify’s CEO, wrote in an article on X that a skill someone wrote to teach River about the checkout data warehouse “gets reused by twelve other teams.” At Wealthsimple, the marketing, client experience, communications and data teams have written Compass skills of their own. And at both companies, for now, a person reviews a change before it goes in.

The obvious objection is that every one of these parts is common this year. Grok Bot, the platform my bots run on, ships the same ones. I talk to a bot in a chat, it gets its abilities from skills, and it can stop and ask me before it acts. So the overlap mostly shows what this year’s tools can do.

What each company insisted on

Shopify insisted that River work where others can see it. River takes no direct messages. In Tobi’s words, it “politely declines” and suggests a public channel instead. Every conversation becomes a Slack thread that other employees can open, and Shopify’s engineers say the next person with a similar problem “starts from this thread.” River can pick up from that thread later too.

Wealthsimple wants people to see the work too. In a write-up of an interview published by the investment firm Sagard, Wealthsimple’s CTO, Diederik van Liere, said they encourage people to work in public “so others can learn by watching.” Compass answers in any channel it has access to. The company’s post points to #building-in-public, a channel everyone at Wealthsimple can view. In the post’s example, someone asks Compass to debug a failing service, and a person from another team joins in with the context it needs. When Compass answers a question about the company’s data, it also shows the query it ran, “so you can check its work.” Both companies value people seeing the agent at work, and Shopify is the one that made it a rule.

Where a change goes in, the two companies are heading different ways. At Shopify, people still review what River writes. Tobi wrote that about one in eight pull requests merged the week before he wrote was “authored by River, reviewed by us.” In River’s security work, the prompts and skills it runs on spell out “human merge authority.” In the Sagard write-up, van Liere said Wealthsimple is “actively experimenting” with letting low-risk pull requests that pass every check go in without human review. That experiment is about pull requests across the company, whoever wrote them.

These choices decide who sees an agent’s work before others rely on it, and what they can check when they see it.

The bots I run

At Shopify and Wealthsimple, many people share one agent, and they talk to it in channels where others can read along. My team is the other way around: one person and a few bots. I talk only to my chief of staff. We discuss the broad direction, and then it tells the bot concerned what to do. (It is a different chief of staff from the one in What It Takes to Win the Personal Agent Market, which runs on my own machine.)

Lauren Tan works on Grok Bot, and on October 31 she posted a short guide to how she builds bots and manages her team, with more tips in a reply. One tip is to share your bot in Slack if you work on a team plan, so a bot can sit in channels as River and Compass do. My bots have only me, and most of her guide fits a team run by one person. While you teach a bot a task, she suggests telling it to write a skill for itself, so it does the task more consistently, even over a long conversation. Routines let a bot act on a schedule, or when something happens, without waiting for a message. A bot’s skills and scripts can also go into a private repository.

My bots have repositories too. Each bot has a repository of its own, which holds that bot’s context. Its skills and scripts are there, along with its logs and the files it keeps to track its work and what came before. Right now everything goes into those repositories, and some of it might sit better in a separate database, which is a question for another time. Wherever it is kept, what one bot learns reaches another bot only if someone passes it on.

Learning from their own work

At Shopify, besides the open threads, River also looks back on its day. On The Knowledge Project podcast, Tobi described how, periodically, at night or in off hours, River is handed the day’s conversations, asked what went well and what it struggled with, and writes what it learned into its skills and instructions. Nobody has to ask for it each time. At Wealthsimple, engineers assign Compass tickets on its own code, and a person reviews each change it writes.

My bots also review their own work on a schedule. When a lesson from one bot could help another, my chief of staff proposes passing it on, and it goes only if I agree. Sometimes I suggest it myself. This happens a few times a week, and so far every bot that received a lesson did its job better afterward.

Passing lessons this way leaves a gap. At Shopify, one skill written for River is reused by twelve other teams, and other employees can read the threads River works in. If a lesson there is wrong, someone besides its author has a chance to catch it. In my team, only my chief of staff and I see a lesson before it reaches another bot, and we see only the ones that come up while we talk. A lesson that a bot’s own review finds, and that neither of us brings up, stays with that bot.

Lauren’s favorite tip ends with a check that covers part of this. After a brain dump by voice, she tells the bot to “restate this in your own words” so she can confirm it understood her. My chief of staff already does this with me. Before it acts, it restates what I told it, and I confirm it got it right. The check stops there. After that, it tells the other bot, and I never see the words that bot receives. When I agree to a lesson, I have seen only a short description of it.

A weekly list of lessons

Starting this week, my chief of staff gathers the lessons from the bots’ own reviews once a week and sends them to me as one list, on top of the ones that come up in our chat. Each item names the bot the lesson would go to and the exact words that bot would keep from now on. I answer yes or no, item by item.

The list runs on a schedule, like River’s look-back, so a lesson no longer waits for one of us to bring it up. Each item shows the exact words, much as Compass shows the query it ran, so I can check the work myself. My chief of staff already restates what I tell it, the way Lauren suggests. With the list, I also see what it tells the other bot.