Semantic Search using Datasette & Python

blaise · December 22, 2023, 4:11am

josephrmartinez/recipe-dataset/blob/main/tutorial.md

# Implement Semantic Search using Datasette
### Spinning up 'vibe-search' on a SQLite Database

[Datasette](https://datasette.io/) is an open-source tool for exploring and publishing data. It allows you to create web interfaces for exploring databases with minimal effort. Developed by Simon Willison, a co-creator of the Django web framework, Datasette is similarly designed to be simple, lightweight, and easy to use. One of the things it greatly simplifies is the ability to quickly spin up "semantic search" against a SQLite database. 

Semantic search is a method for returning highly relevant results based on *meaning* rather than just keyword filtering. This allows users to find documents that may relate to their search term, even when the matching keywords are not present in the search query. This is ideal for those instances where we are trying to search in a general direction or where we don't yet know the right keywords to use.

I discussed this with a lawyer friend of mine and he immediately understood it as "vibe search." Fittingly, this is the same term that Willison used in his [excellent post](https://simonwillison.net/2023/Oct/23/embeddings/) that describes working with semantic search and embeddings in more detail. 

I'm going to walk through how I used Datasette to quickly create and deploy a functional semantic search engine on a database of 5,000 recipes. With this functionality, we will be able to search our database for "a small levantine appetizer" and receive highly useful results. That is an odd way to look up a recipe but semantic search "just gets" what we mean.

The key point here is that you do not *need* a vector database or lots of tooling to quickly spin up powerful semantic search on a decent-sized data set.

*Note: this tutorial is largely modeled after Willison's posts on his blog, starting with [this post](https://simonwillison.net/2023/Jan/13/semantic-search-answers/). I simplified things a bit and focused more narrowly on implementing semantic search and working with Datasette for the first time.* 

## Tutorial

### Download dataset into project directory
We're going to work with a simple dataset that includes the title, ingredients, and instructions for 5,000 recipes. Grab the 5k-recipes.db file from this [recipe dataset repository](https://github.com/josephrmartinez/recipe-dataset).

This file has been truncated. show original

blaise · December 22, 2023, 4:13am

This is largely modeled after Simon’s post How to implement Q&A against your documentation with GPT3, embeddings and Datasette but in this tutorial I’m providing the user with a dataset and walking them through what it is like to work with Datasette for the first time.