In this guide, we're going to build a Discord bot that helps you manage an ever-growing knowledge base.

DeepStructure's Discord server has a #relevant channel where our team posts links. We dogfood a version of this bot internally to process the links posted in #relevant. Without the bot, we would never be able to read all the content, or answer our questions about the older links!

The bot will:

• Ingest and index links that you post in a Discord channel, to grow your knowledge base
• Summarize the content of the links and reply in thread
• Respond to questions in Discord with answers drawn from the knowledge base (i.e. RAG)
• Expose HTTP endpoints to directly ingest documents and answer questions, from any of your other existing systems

🔑 Get your API key

Currently, DeepStructure is accepting new developers by invitation only. Please drop us a line at [email protected]!

Once you have your key, install the CLI and provision your system with the following commands:

npm install -g @deepstructure/cli
ds login

📝 Create a basic RAG app

Run the following commands:

# Prefer to use TypeScript with the --ts switch
npx create-deepstructure-app --ts knowledge-bot
cd knowledge-bot

Now you've created a directory structure that looks like the following:

knowledge-bot/
  src/
  	app.ts
		rag.ts
	.env.example
	README.md
	.gitignore
	package.json
	tsconfig.json

This should be familiar to you from other TypeScript / npm / node apps you've worked with. app.ts contains your Application code, most important of which are your workflows. You don't need to dive into the details of those yet.

However, feel free to begin editing your app.ts at any time!

🏗️ Run your app locally

Currently, local development is only available for Enterprise partners.

🚢 Deploy your app to production

Just takes a couple of additional commands:

ds app create
ds deploy

After ds deploy succeeds, it will print the URL of the Web Console for your app. Navigate to the Web Console to see summary stats of your app, errors and failures that require your attention, and the list of all workflow runs with component data flows and log messages.

📘

The Web Console is your best friend in understanding and debugging the live behavior of your deployed app.

The Web Console also allows you to send requests directly to all HTTP endpoints exposed from your app, via an embedded Swagger panel. Alternatively, you can experiment with the endpoints via curl, wget, or your favorite API exploration tool.

Your app is now ready to ingest and index documents, and answer questions about the dynamically-evolving document corpus!

🤖 Add a user interface (Discord)

Before we add the Discord interface to our app, let’s go into a bit more detail about the DeepStructure design philosophy and what makes it different:

• DeepStructure components conceptually take input from SQL tables, transform the values, then write the transformed output back to SQL tables
• Data Connectors can “lift” external data — like Discord messages — into the DeepStructure domain of SQL tables
• And Data Connectors can also “lower” SQL tables back into external formats — like Discord messages

But we believe that copying & pasting from a working application is one of the best development methods. Here's a snippet of code from our reference bot app, which you're welcome to mash up and customize for your own apps. This workflow ingests links from Discord messages that @-mention the bot:

//...
const saveLinkWorkflow = new DiscordSource({
    displayName: "Discord_Message_Capture",
})
    .pipe(botOutputTable)
    .filter(
        (value: DiscordMessageDBValue) =>
            value.channelId === process.env.DS_DISCORD_CHANNEL_ID
    )
    .pipe(botOutputFiltered)
    .pipe(
        new RowTransformComponent({
            transformerName: "URL_EXTRACTOR",
        })
    )
    .pipe(linksTable)
    .withName("saveLink");
//...

(We've omitted the details here of creating and authorizing a new application for your Discord server, and setting the necessary environment variables and secrets. You can use the ds CLI to set environment variables and secrets for your app.)

When you're ready, re-deploy your app with ds deploy. You'll be in for a surprise: the bot will backfill old messages from your Discord into the indexing pipeline! (If you enable that behavior.) This comes naturally from the way that DeepStructure Data Connectors model the external world.