I'm building Second Memory, a personal memory application that allows users to save their memories and retrieve them later through natural-language questions.

One challenge is that users don't necessarily remember the exact words they used when saving a memory. They might remember its context or meaning instead.

For example, a user might have saved these memories:

• "I forgot my umbrella and got soaked in the rain."

• "I lifted weights at the gym after work."

• "I caught up with John over coffee yesterday."

• "I stayed up late debugging my application."

Later, the user asks:

"Who did I meet recently?"

The most relevant memory would be:

"I caught up with John over coffee yesterday."

A traditional keyword search might miss this because "meet" and "caught up" are different expressions.

This is where semantic search comes in. Instead of relying on exact keyword matches, we can search for text with similar meanings using embeddings.

In this post, I'll explore how to generate embeddings, calculate cosine distance, find relevant sentences, and perform vector searches using PostgreSQL with pgvector.

The examples use a locally runnable embedding model, so no API key or paid embedding service is required.

Generate Embeddings

An embedding is a numerical representation of text. An embedding model converts text into a vector, allowing us to compare sentences mathematically.

For this demonstration, I'll use sentence-transformers/all-MiniLM-L6-v2, a small, open-source embedding model that runs locally and produces 384-dimensional vectors.

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")

sentences = [
    "I forgot my umbrella and got soaked in the rain.",
    "I lifted weights at the gym after work.",
    "I caught up with John over coffee yesterday.",
    "I stayed up late debugging my application.",
]

embeddings = model.encode(sentences)

print(embeddings.shape)
# (4, 384)

print(embeddings)
# [[ 0.04035296  0.10288464  0.09306896 ... -0.00502765 -0.13130508  0.05253862]
#  [-0.07009698  0.04319851  0.00160278 ... -0.06155823 -0.09415536 -0.01941661]
#  [-0.04802262 -0.01395144  0.12706237 ... -0.0639021  -0.03755342 -0.06052431]
#  [ 0.00379441  0.01192298 -0.02876158 ... -0.02021407 -0.0892337   0.05982817]]

The model converts each sentence into a vector containing 384 floating-point values.

The shape (4, 384) represents four sentences, each with a 384-dimensional embedding.

Each number is a coordinate in a high-dimensional vector space. Individual dimensions don't necessarily correspond to specific concepts such as people, places, or activities. Instead, semantic information is distributed across the vector.

The output is abbreviated with ..., but each embedding contains all 384 values.

Looking at these numbers alone doesn't tell us which sentences are semantically related. To find that out, we need to compare their vectors.

Using an Embedding API

Embeddings can also be generated through an API rather than running a model locally.

For example, OpenAI provides an embeddings endpoint:

curl https://api.openai.com/v1/embeddings \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "text-embedding-3-small",
    "input": "I caught up with John over coffee yesterday."
  }'

The response contains an embedding vector, just like the local model.

The concept is the same, although different models produce different vector dimensions and values. Embeddings generated by different models generally shouldn't be mixed in the same similarity search.

For the rest of this post, I'll continue using the local model so the examples remain reproducible without an API key.

Calculate Cosine Distance

Now that we have embeddings, we need a way to measure how closely two sentences are related.

One commonly used metric is cosine distance, which compares the directions of two vectors rather than their magnitudes.

Let's calculate the distance between two sentences.

import numpy as np
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")

embeddings = model.encode([
    "I forgot my umbrella and got soaked in the rain.",
    "Who did I meet recently?"
], normalize_embeddings=True)

# Cosine distance using normalized vectors
distance = 1 - np.dot(embeddings[0], embeddings[1])

print(f"Cosine distance: {distance:.4f}")
# Cosine distance: 0.7981

The normalize_embeddings=True option scales each vector to unit length.

For normalized vectors, the dot product (np.dot()) is equivalent to cosine similarity. Subtracting it from 1 gives us cosine distance:

$$\text{cosine distance} = 1 - \text{cosine similarity}$$

Sentence Transformers also provides model.similarity(), which calculates cosine similarity by default. The equivalent calculation would be:

distance = 1 - model.similarity(
    embeddings[0:1], embeddings[1:2]
).item()

However, using NumPy keeps the distance calculation independent of Sentence Transformers. The same approach works with embeddings returned by OpenAI or other embedding APIs.

A smaller cosine distance indicates greater similarity. A distance of 0 means the vectors point in the same direction.

Why Cosine Distance?

There are several ways to compare vectors:

For text embeddings, we're generally interested in semantic relationships rather than differences in vector magnitude, making cosine distance a practical default.

When embeddings are L2-normalized, cosine distance, Euclidean distance, and inner product produce equivalent rankings. The choice of metric matters more when embeddings aren't normalized or when the model recommends a particular metric.

We'll use cosine distance throughout this post because pgvector supports it directly through the <=> operator.

Find the Most Relevant Memory

Now let's apply cosine distance to our original example and find the memory most relevant to the question:

"Who did I meet recently?"

import numpy as np
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")

texts = [
    "I forgot my umbrella and got soaked in the rain.",
    "I lifted weights at the gym after work.",
    "I caught up with John over coffee yesterday.",
    "I stayed up late debugging my application.",
]

query = "Who did I meet recently?"

memory_embeddings = model.encode(texts, normalize_embeddings=True)
query_embedding = model.encode(query, normalize_embeddings=True)

# Cosine distance using normalized vectors
distances = 1 - np.dot(memory_embeddings, query_embedding)

for i in np.argsort(distances):
    print(f"{distances[i]:.4f} - {texts[i]}")
# 0.6097 - I caught up with John over coffee yesterday.
# 0.7981 - I forgot my umbrella and got soaked in the rain.
# 0.8151 - I lifted weights at the gym after work.
# 0.8454 - I stayed up late debugging my application.

This time, we're comparing a single query against multiple candidate memories.

The results are sorted in ascending order because smaller cosine distance indicates greater relevance.

As expected, the memory about meeting John ranks first, even though the question uses "meet" while the memory uses "caught up".

This demonstrates semantic retrieval: finding related meanings rather than matching exact words.

However, the closest result isn't guaranteed to be relevant in every case. The ranking depends on the embedding model and the available memories.

In a production system, we'd need to evaluate retrieval quality using representative queries and data.

Introducing pgvector

So far, we've generated embeddings and calculated distances in Python. But generating embeddings for every stored memory on each query would be inefficient.

A better approach is to generate embeddings when memories are created or updated, persist them, and search against the stored vectors.

For this, I'll use pgvector, a PostgreSQL extension that adds vector storage and similarity search capabilities.

Before using it with text embeddings, let's look at a simple example with two-dimensional vectors.

Store Vectors

First, enable the extension and create a table:

CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE examples (
    id SERIAL PRIMARY KEY,
    embedding VECTOR(2)
);

Insert three vectors:

INSERT INTO examples (embedding)
VALUES
    ('[1,0]'),
    ('[0.8,0.6]'),
    ('[0,1]');

Using two-dimensional vectors makes it easier to see how pgvector calculates distances.

Search by Cosine Distance

Let's find the vectors closest to [1,0].

SELECT
    embedding,
    embedding <=> '[1,0]' AS distance
FROM examples
ORDER BY distance;

The <=> operator calculates cosine distance.

The results are:

 embedding | distance
-----------+----------
 [1,0]     | 0
 [0.8,0.6] | 0.2
 [0,1]     | 1

The first vector has a distance of 0 because it points in the same direction as the query.

The second vector has a distance of 0.2, while the third is orthogonal to the query and has a distance of 1.

The important point is that pgvector doesn't understand the meaning of text. It stores vectors and performs mathematical operations on them.

The embedding model is responsible for producing vectors that represent semantic relationships.

Vector Indexes

Without an index, PostgreSQL performs an exact nearest-neighbor search.

For larger datasets, pgvector supports approximate nearest-neighbor indexes such as HNSW and IVFFlat.

For example, we can create an HNSW index:

CREATE INDEX examples_embedding_hnsw_idx
ON examples
USING hnsw (embedding vector_cosine_ops);

The search query itself doesn't need to change.

PostgreSQL can use the index to accelerate searches, although the query planner may choose a sequential scan for small datasets.

Approximate indexes can improve search performance by trading off some retrieval accuracy.

Embedding Dimensions and Index Limitations

One practical consideration when using pgvector is the number of dimensions in an embedding.

pgvector supports storing vectors with up to 16,000 dimensions, but standard HNSW and IVFFlat indexes using the vector type support up to 2,000 dimensions.

This becomes relevant when using models such as OpenAI's text-embedding-3-large, which produces 3,072-dimensional embeddings by default.

One option is to use a shorter embedding supported by the model.

OpenAI's text-embedding-3 models use Matryoshka Representation Learning (MRL), which allows shorter embeddings to retain useful semantic information.

For example, we can request a 1,536-dimensional embedding from text-embedding-3-large:

curl https://api.openai.com/v1/embeddings \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "text-embedding-3-large",
    "input": "I caught up with John over coffee yesterday.",
    "dimensions": 1536
  }'

This reduces storage requirements and allows the embedding to fit within pgvector's standard vector index dimension limit.

Another option is pgvector's halfvec type, which supports indexing up to 4,000 dimensions using half-precision values.

The right approach depends on retrieval quality, storage, precision, and performance requirements.

For our example using 384-dimensional embeddings, the standard vector type is sufficient.

Store and Search Embeddings with pgvector

Now that we've covered embedding generation, cosine distance, and pgvector, let's bring everything together.

We'll store the four memories with their embeddings in PostgreSQL and retrieve them using cosine distance.

import psycopg
from pgvector.psycopg import register_vector
from sentence_transformers import SentenceTransformer

DB_URL = "postgresql://user:password@localhost:5432/mydatabase"

model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")

texts = [
    "I forgot my umbrella and got soaked in the rain.",
    "I lifted weights at the gym after work.",
    "I caught up with John over coffee yesterday.",
    "I stayed up late debugging my application.",
]

query = "Who did I meet recently?"

DIMENSIONS = 384

with psycopg.connect(DB_URL) as conn:
    conn.execute("CREATE EXTENSION IF NOT EXISTS vector")
    register_vector(conn)

    conn.execute(f"""
        CREATE TABLE IF NOT EXISTS journal (
            id BIGSERIAL PRIMARY KEY,
            content TEXT NOT NULL,
            embedding VECTOR({DIMENSIONS}) NOT NULL
        )
    """)

    embeddings = model.encode(texts)

    # Reset the demo data
    conn.execute("TRUNCATE TABLE journal RESTART IDENTITY")

    for content, embedding in zip(texts, embeddings):
        conn.execute(
            """
            INSERT INTO journal (content, embedding)
            VALUES (%s, %s)
            """,
            (content, embedding)
        )

    conn.commit()

    query_embedding = model.encode(query)

    results = conn.execute(
        """
        SELECT
            content,
            embedding <=> %s AS distance
        FROM journal
        ORDER BY embedding <=> %s ASC
        LIMIT 4
        """,
        (query_embedding, query_embedding)
    ).fetchall()

    print(f"Query: {query}")

    for content, distance in results:
        print(f"{distance:.4f} - {content}")
# Query: Who did I meet recently?
# 0.6097 - I caught up with John over coffee yesterday.
# 0.7981 - I forgot my umbrella and got soaked in the rain.
# 0.8151 - I lifted weights at the gym after work.
# 0.8454 - I stayed up late debugging my application.

There are two key differences from the in-memory implementation.

1. Embeddings are persisted

pgvector provides the VECTOR(384) data type, allowing embeddings to be stored alongside their original text.

register_vector(conn) integrates pgvector with psycopg so NumPy arrays can be passed directly as parameters.

In a real application, embeddings would be generated when memories are created or updated, rather than on every query.

2. Similarity search moves into PostgreSQL

Instead of loading every embedding into Python to calculate distances, we let PostgreSQL perform the search.

The <=> operator calculates cosine distance, and the query returns memories ordered by their distance from the query embedding.

As shown in the output, the ranking matches our earlier Python example, with the memory about meeting John appearing first.

For larger datasets, vector indexes can improve search performance without changing the core retrieval logic.

Conclusion

We've explored how embeddings represent sentence meanings numerically, how cosine distance helps identify related text, and how to retrieve relevant sentences even when their wording differs.

We started with a locally runnable embedding model, calculated distances in Python, and then introduced pgvector using simple two-dimensional vectors.

Finally, we combined everything into a PostgreSQL-backed semantic search implementation.

The same retrieval approach can be used in a Retrieval-Augmented Generation (RAG) system, where retrieved text is provided to an LLM as context for generating an answer.

There are further considerations for production systems, including vector indexing, hybrid search, relevance thresholds, and retrieval evaluation.

But the core mechanism is straightforward: generate embeddings, calculate distances, and retrieve the closest matches.

The goal of Second-Memory is to help users capture their thoughts and retrieve meaningful information from them. Building an authentication system doesn't directly advance that goal.

There's also a user-experience consideration. People already manage many online accounts. Asking them to create and remember another password adds friction, while social login lets them sign in with an account they already have.

Build vs. Buy

I considered two approaches: building authentication in-house or using an existing service.

For a project like Second-Memory, building authentication from scratch would mean taking on additional complexity without providing much differentiation.

What Does In-House Authentication Involve?

Building authentication is more than verifying a username and password. It means taking responsibility for several areas.

1. Security

• Store passwords using an appropriate password-hashing algorithm.
• Protect against brute-force attacks and credential stuffing.
• Implement secure session management, including expiration and revocation.
• Protect account recovery and verification flows against account takeover.
• Manage secrets, security logging, monitoring, and incident response.

2. Account lifecycle

• Registration and email verification
• Login and logout
• Account recovery and password resets
• Account deletion and related data cleanup
• Session revocation and suspicious-login handling

3. Social login integration

Supporting social login involves integrating with external identity providers and handling their authentication flows. Standards such as OpenID Connect (OIDC) and OAuth 2.0 are commonly involved, depending on the provider and flow.

The application must also handle identity mapping, account linking where appropriate, and the relationship between an external identity and its own user records.

4. Ongoing operations

Authentication requires continued attention after launch. Security vulnerabilities emerge, dependencies need updates, and authentication flows must be tested as the application evolves.

The effort doesn't end when login starts working.

The Decision

I don't have specific authentication requirements that justify building and maintaining my own system.

I'll use Firebase Authentication and make social login the preferred way to sign in. This avoids asking users to create another password and lets me rely on an established authentication service.

There may be usage costs and some provider dependency, but building authentication in-house also has costs in development time, maintenance, and security responsibilities.

For Second-Memory, Firebase Authentication is a practical choice because it meets my needs without requiring me to build and operate an authentication system myself.

However, using a third-party authentication service doesn't eliminate application security responsibilities. Second-Memory still needs to validate identities correctly, enforce authorization in the backend, and ensure users can access only their own data.

How Authentication Works in Second-Memory

Second-Memory has web and mobile clients, plus backend services. Both clients use Firebase Authentication, while the backend verifies each request and maps the Firebase identity to an internal user record.

The authentication flow

The flow has six steps:

1. Sign in: The client signs in with Firebase Authentication and obtains a Firebase ID token.
2. Call the API: The client sends the token in the Authorization: Bearer <token> header.
3. Verify the token: The backend verifies the token using the Firebase Admin SDK. It must not trust a token simply because it can decode it.
4. Extract the Firebase UID: After verification, the backend reads the user's Firebase UID from the token.
5. Resolve the internal user: The backend looks up the user record using firebaseUid.
6. Authorize the request: The backend uses the internal user ID to enforce access controls and scope data operations to that user.

Why maintain our own user table?

Firebase provides authentication, but Second-Memory still needs its own application-level user record.

For example:

The id is Second-Memory's internal user ID, while firebaseUid links the record to Firebase Authentication. The firebaseUid column should have a unique constraint.

This separation has a few advantages:

• Domain independence: Memories and other application entities can reference Second-Memory's internal user ID.
• Consistent authorization: Backend services can use the same internal identity when checking data access.
• Flexibility: If the authentication provider changes in the future, the application's domain model doesn't need to use provider-specific identifiers everywhere.

The backend must still enforce authorization on every relevant request. A valid Firebase token proves the caller's authenticated identity; it does not automatically grant access to a particular memory or resource. The backend should derive identity from the verified token, not trust a user ID supplied by the client.

How this fits the client and backend stack

The same authentication principle applies across the different parts of Second-Memory, although their implementations differ.

• Next.js web app: Configure Firebase for the web client, implement sign-in, and attach the ID token to API requests.
• Expo mobile app: Configure Firebase Authentication for React Native, handle session persistence, and attach the ID token to API requests.
• Backend services: Use the Firebase Admin SDK to verify ID tokens, resolve the internal user, and enforce authorization.

The important distinction is that the clients obtain tokens, but the backend is responsible for verifying them and deciding what the authenticated user is allowed to access.

Conclusion

For Second-Memory, authentication is necessary infrastructure, not a feature I need to reinvent.

Firebase Authentication reduces the amount of authentication code I need to build and maintain. Keeping an internal user record also lets Second-Memory separate external identity verification from application-level identity and authorization.

The goal isn't to eliminate security responsibilities. It's to use an established service for authentication while keeping control of the application's own data and access rules.

A monorepo is a single repository containing multiple distinct projects, with well-defined relationships. — Monorepo Explained

A monorepo makes sense when multiple projects have meaningful relationships:

• Shared libraries and utilities — reuse common code across projects.

• Shared contracts — keep API types consistent between producers and consumers.

• Cross-project changes — update related projects in a single change.

• Shared tooling and rules — maintain consistent development practices.

If projects have no meaningful relationships, separate repositories may be simpler.

For Second-Memory, Web, Mobile, and Backend need to agree on API contracts. Sharing TypeScript API types helps keep those contracts consistent.

Web uses React/Next.js, while Mobile uses React Native/Expo, so their UI implementations remain separate. Code is shared only where it genuinely works across both platforms.

Keeping related projects together makes coordinating changes and managing shared contracts easier.

How do we manage their packages and dependencies?

Why Workspaces?

Workspaces are a set of features in the npm CLI that provide support for managing multiple packages from a local file system within a single top-level root package. — npm Workspaces

A monorepo can work without workspaces. Each project can have its own package.json and manage dependencies independently.

For example:

second-memory/
├── apps/
│   ├── web/
│   └── mobile/
├── services/
│   ├── memory-service/
│   ├── ask-service/
│   └── embedding-worker/
└── packages/
    ├── api-types/
    └── ui-component/

Suppose both Web and Memory Service depend on api-types.

Without workspaces, each project needs to manage its dependencies and local references separately. A project might reference the shared package using a local file dependency:

{
  "dependencies": {
    "@second-memory/api-types": "file:../../packages/api-types"
  }
}

The relative path depends on the consuming project's location. Dependencies must be installed and managed in each project, and keeping local references and updates consistent requires additional care.

With workspaces, the root configuration defines which directories contain packages.

For example, with pnpm:

pnpm-workspace.yaml

packages:
  - "apps/*"
  - "services/*"
  - "packages/*"

Each package declares its own dependencies in its package.json. For example, Web can depend on the shared API types package:

apps/web/package.json

{
  "dependencies": {
    "@second-memory/api-types": "workspace:*"
  }
}

The shared package identifies itself in its own package.json:

packages/api-types/package.json

{
  "name": "@second-memory/api-types",
  "version": "1.0.0"
}

The workspace:* protocol explicitly tells pnpm to use the local workspace package.

Dependencies can then be installed from the repository root:

pnpm install

pnpm recognises the workspace packages and links local dependencies, making it easier to manage relationships between projects.

Without Workspaces vs With Workspaces

A workspace reduces manual package and dependency management. It provides a way to define packages and their local dependencies within the monorepo.

However, managing packages is only part of the problem. As the number of projects and dependencies grows, coordinating builds, tests, and other tasks becomes another challenge.

Why Task Orchestration?

With only a few projects, running build and test commands manually may be enough. As the repository grows, two problems become more noticeable.

Problem 1: Running Only the Necessary Tasks

The dependencies in Second-Memory look like this:

Suppose I change packages/ui-component.

Web and Mobile depend on it, but Memory Service, Ask Service, and Embedding Worker do not.

Ideally, I want to run checks for the affected projects without wasting time checking unrelated services.

With workspaces alone, I can run individual commands using package filters:

pnpm --filter web test
pnpm --filter mobile test

But I still need to know which projects depend on the changed package and decide which commands to run.

As the dependency graph grows, maintaining this selection manually becomes more difficult.

Task orchestration can automate the selection of relevant tasks based on package relationships, configured task dependencies, and changes.

Problem 2: Managing Dependencies Between Tasks

Some tasks need to run before others.

For example, if a service consumes a shared package's compiled output, that package needs to be built before the service.

I could maintain a script to enforce the order:

{
  "scripts": {
    "build:memory": "pnpm --filter api-types build && pnpm --filter memory-service build"
  }
}

This works for a simple case, but the scripts become harder to maintain as more packages and task dependencies are added.

Ideally, I want to define task relationships once and let a tool determine the correct execution order.

How Task Orchestration Helps

Task orchestration adds a layer above workspace package management. It coordinates tasks such as builds, tests, and linting across packages.

Task Graph

A task graph describes dependencies between tasks and helps determine their execution order.

For example, if api-types must be built before its consumers:

The graph should reflect the actual build requirements. If consumers compile the shared package's source directly, a separate build step may not be necessary.

Turborepo uses the workspace package dependency graph together with configured task dependencies to determine execution order. The package relationships provide the foundation; task configuration defines how tasks depend on one another.

Affected Tasks

When a package changes, Turborepo can use Git change information and the dependency graph to select affected packages and their relevant tasks.

• Change api-types: Web, Mobile, Memory Service, and Ask Service may be affected.

• Change ui-component: Web and Mobile may be affected.

• Change embedding-worker: Its checks can run independently of the other projects.

This reduces the need to maintain manual build and test sequences as the repository grows. The result depends on the configured task graph, available Git change information, and the tasks defined by each package.

Additional Benefits

These capabilities can improve development and CI performance, although the primary goal is to manage task relationships and execution more reliably.

Comparing the Approaches

Workspaces alone can be sufficient for a small repository with simple build and test requirements. Turborepo and Nx become useful when coordinating tasks across project dependencies requires more automation.

Both provide task orchestration, affected-task capabilities, caching, and parallel execution. Nx offers a broader set of project tooling and plugins, while Turborepo focuses more directly on task orchestration.

For Second-Memory, I wanted the task orchestration capabilities without needing a broader project-management system.

How should I organise the codebase?

I chose a monorepo.

Second-Memory has several applications and services:

apps/
  web/
  mobile/

services/
  memory-service/
  ask-service/
  embedding-worker/

packages/
  api-types/
  ui-components/

They are independently deployable, but they are all part of the same product.

Monorepo vs Polyrepo

There is no universally better choice.

The trade-offs become clearer when looking at common development tasks:

Neither approach is inherently better.

For Second-Memory, I valued easy development across the whole system more than repository-level isolation.

When to Choose Which

A monorepo can be a good fit when:

• Small team or solo developer — one repository is easier to manage than several.

• Projects share contracts or code — changes can be made and tested together.

• Cross-project changes are common — one PR can update multiple applications or services.

• The whole system is closely related — one repository makes it easier to understand and work across the product.

Polyrepo can be a better fit when:

• Large or independent teams need clear ownership boundaries.

• Strong repository isolation is important.

• Projects have different release cycles.

• Access control needs to differ between projects.

For Second-Memory, being a sole developer was one of the main reasons I chose a monorepo.

Managing several repositories would add overhead without giving me much benefit. I didn't need repository-level ownership boundaries between teams.

Instead, I wanted to make it easy to work across the web app, mobile app, backend services, and shared contracts.

What Should Be Shared?

A monorepo makes sharing easy.

But easy to share doesn't mean everything should be shared.

I wanted to keep the shared surface area small.

So instead of creating shared packages for everything, I only create a package when there is a genuine reason to share something.

For example:

apps/
  web/
  mobile/
services/
  memory-service/
  ask-service/
  embedding-worker/
packages/
  api-types/
  ui-components/

The API types are shared because they represent a contract between different parts of the system.

But a Web-only component stays in apps/web.

A Mobile-only utility stays in apps/mobile.

And Memory Service domain logic stays inside memory-service.

Share the contract, not the ownership

The fact that services live in the same repository doesn't mean they should share their internal implementation.

The API contract can be shared:

while the services remain independently owned:

Share contracts where it makes sense, but don't share service ownership.

Keeping the Shared Surface Small

There is another practical reason for limiting shared code.

The more projects depend on a shared package, the larger the potential impact of a change.

For example:

A change to shared may require all projects to be tested.

But a change here:

only affects Web.

This has implications for both CI/CD and application dependencies.

I don't want a large shared package that contains unrelated functionality used by almost every project.

Instead, I prefer small, focused shared packages with clear reasons for existing.

The CI/CD Trade-off

CI/CD is one area where polyrepo can be simpler.

With polyrepo, a change to memory-service naturally belongs to the memory-service repository.

In a monorepo, multiple deployable units live in the same repository.

A change to:

services/memory-service/

shouldn't necessarily rebuild and deploy everything.

The pipeline needs to understand:

• what changed

• what depends on it

• what needs to be tested

• what needs to be built

• what needs to be deployed

This becomes particularly important when shared packages are involved.

A change to ui-components may affect only web and mobile projects, while a change isolated to services/embedding-worker/ should ideally affect only the relevant pipeline.

The repository structure itself is simple, but the build and deployment system needs to become more intelligent as the project grows.

Why Monorepo for Second-Memory?

The main benefits for me were:

• Easy sharing of API types

• Easy cross-project changes

• Easy local development

• Easy dependency management

• Easy visibility of the whole product

• Less overhead for a solo developer

The trade-offs were:

• More complex CI/CD

• More tooling needed as the repository grows

• Less repository-level isolation

• The need to carefully control shared dependencies

For Second-Memory, I was willing to accept these trade-offs.

The goal wasn't to share as much code as possible.

The goal was to get the convenience of a monorepo while keeping ownership and dependencies clear.

That led to the next question:

How do I manage builds, dependencies, task execution, and caching across all these projects?

• Memory Service

• Ask Service

• Embedding Worker

But there was another architectural decision behind this:

Why make them separate services instead of building one backend?

A monolith would have been simpler.

For a small application, it would probably have been enough.

I still chose a microservice architecture for Second-Memory for three main reasons.

1. Clear boundaries

Memory and Ask have different responsibilities.

The Memory Service owns the memory domain:

• create and manage memories

• persist memory data

• generate/retrieve relevant memories

• manage embeddings

The Ask Service owns the AI interaction:

• receive questions

• retrieve relevant context

• construct prompts

• call the LLM

• return the answer

They work together, but they represent different responsibilities.

I wanted those boundaries to exist at the service level rather than only as modules inside one application.

The Embedding Worker is separated for a different reason.

Embedding generation is asynchronous work and doesn't need to be part of the request that creates a memory.

The boundaries therefore follow the responsibilities and execution models of the system.

2. A system for learning AI engineering

Second-Memory is also a project for exploring how AI systems are built.

I didn't want the architecture to stop at:

The system gave me an opportunity to work with several patterns that are common in production AI systems:

• asynchronous processing

• queues

• workers

• embeddings

• vector search

• retrieval

• service-to-service communication

• LLM integration

Using separate services makes these boundaries explicit and gives me a way to experiment with each part independently.

The goal isn't to use microservices simply because they are popular.

The architecture should help me learn and validate how these pieces work together as a real system.

3. Production readiness and scaling

I also wanted to design Second-Memory as something that could eventually grow into a production system.

The different components have different scaling characteristics.

If memory creation increases, I may need more Memory Service instances.

If embedding jobs increase, I can scale the workers independently.

If AI requests become the bottleneck, the Ask Service can scale independently.

This doesn't mean I need to scale everything independently today.

It means the architecture doesn't force unrelated workloads into the same deployment unit.

What about a monolith?

A monolith would still be a reasonable choice for Second-Memory.

It would have:

• less infrastructure

• simpler deployment

• simpler local development

• fewer service-to-service calls

• less operational overhead

So why accept the additional complexity?

Because my goal wasn't only to build the simplest V1.

I wanted to build a system with clear boundaries that could evolve toward production scale, while giving me a practical environment to explore AI system architecture.

The trade-off was intentional.

I chose microservices not because a monolith couldn't work, but because the boundaries, asynchronous workloads, learning goals, and potential scaling characteristics made the additional complexity worthwhile.

The previous post covered why I chose pgvector for semantic search.

But there was another question:

When should a memory be embedded?

At first, it might seem simple:

Create memory
     ▼
Generate embedding
     ▼
Save embedding

But this makes memory creation dependent on the embedding process.

If the embedding provider is slow or unavailable, creating a memory could also fail or become slow.

I wanted to separate these two operations.

The architecture became:

Memory Service
      ▼
 PostgreSQL
      ▼
 Outbox Table
      ▼
 Queue / Job Runner
      ▼
 Embedding Worker
      ▼
 pgvector

1. The Outbox Pattern

When a user creates a memory, the Memory Service writes the memory and an outbox event in the same database transaction.

For example:

Transaction
┌─────────────────────────────┐
│ INSERT memory               │
│ INSERT outbox event         │
└─────────────────────────────┘

The important part is that they succeed or fail together.

This avoids a common problem with distributed systems:

Memory saved
     ▼
Embedding event lost

The outbox gives me a durable record of the work that needs to happen.

The outbox is not the queue itself.

It is a reliable bridge between the database transaction and asynchronous processing.

2. The Queue / Job Runner

A separate process reads pending outbox events and puts jobs onto a queue.

Outbox
   ▼
Queue
   ▼
Embedding Worker

This gives the embedding process some useful properties:

• asynchronous processing

• retries

• independent scaling

• failure isolation

• no need to block the memory API

The user can save a memory without waiting for the embedding provider.

3. The Embedding Worker

The embedding worker has one main responsibility:

turn memory text into an embedding and store it in pgvector.

This keeps embedding-specific logic out of the Memory Service's synchronous request path.

It also gives me a place to evolve the embedding pipeline later.

For example, I could change:

• embedding models

• chunking strategy

• retry behaviour

• batch processing

• embedding dimensions

without changing the API used to create a memory.

4. Why asynchronous?

There is an important consequence of this design:

A newly created memory may not be immediately searchable.

There can be a small delay between:

Memory created
     │ asynchronous processing
     ▼
Embedding created
     ▼
Memory available for semantic search

I accepted this trade-off.

For Second-Memory, immediate persistence is more important than making the embedding operation part of the user's request.

This is essentially eventual consistency between the memory record and its vector representation.

5. Failure becomes easier to handle

The asynchronous design also changes how failures work.

If the embedding provider temporarily fails:

Memory
  ▼
Outbox
  ▼
Queue
  ▼
Embedding Worker - X -> Retry

The memory itself has already been safely stored.

The embedding job can be retried without asking the user to submit the memory again.

That separation was important to me.

The decision

The final flow became:

Each part has a different responsibility:

This added some complexity compared with simply generating the embedding inside the API request.

But it gave me something more important:

memory creation is no longer tightly coupled to the availability of the embedding pipeline.

That was the trade-off I wanted.

Second-Memory stores things a user writes over time.

If a user later asks:

“What did I write about performance problems?”

A normal keyword search isn't always enough.

The memory might say:

“The database becomes slow when thousands of users query it at the same time.”

There may be no exact keyword match between the question and the memory.

This is where semantic search becomes useful.

1. From text to vectors

I can convert a piece of text into an embedding — a vector representation of its meaning.

For example:

"The database becomes slow when thousands of users query it at the same time."
         ↓
[Embedding model]
         ↓
[0.021, -0.183, ...]

The actual vector contains many dimensions, so it isn't meaningful to look at the individual numbers.

What matters is the relationship between vectors.

Texts with similar meanings tend to have vectors that are closer together.

So when a user asks a question, I can:

1. Question

2. Embedding

3. Vector similarity search

4. Relevant memories

This gives Second-Memory a way to retrieve memories based on meaning, rather than just matching words.

2. Why pgvector?

Once I decided to use embeddings, I needed somewhere to store and search them.

I considered:

I chose pgvector.

The main reason wasn't that pgvector is necessarily better than Pinecone.

It was that Second-Memory already had a natural place for the vectors:

the Memory Service's database.

With pgvector, I could keep the memory and its embedding together.

That kept the architecture simple.

3. Why not introduce a vector database?

A dedicated vector database could make sense at larger scale.

But introducing one also creates another system to operate and another boundary to manage.

For V1, I didn't see enough benefit to justify that complexity.

The Memory Service could own:

• memory data

• embeddings

• vector search

all within the same data boundary.

That also reinforced one of the architectural principles from the previous post:

The service that owns the data should own access to it.

The Ask Service doesn't need to know whether semantic search is implemented with pgvector, Pinecone, or something else.

It simply asks the Memory Service for relevant memories.

4. What this gives me

With pgvector, the basic retrieval flow becomes:

User question
      ↓
Generate query embedding
      ↓
Memory Service
      ↓
pgvector similarity search
      ↓
Relevant memories
      ↓
Ask Service
      ↓
     LLM

This is the first important piece of the AI architecture.

The LLM doesn't need to know everything the user has ever written.

Instead, the system retrieves the memories that are most relevant to the current question and uses those as context.

The decision

For Second-Memory V1, I chose:

PostgreSQL + pgvector

because it gave me semantic search without introducing another database.

It was a pragmatic choice:

• vector search

• relational data

• one data boundary

• less infrastructure

• simpler development

Now it was time to design the system.

I started with the main responsibilities of Second-Memory and came up with my first architecture draft:

The diagram itself is fairly simple, but there were several design principles behind it.

1. BFF for the clients

Both the web and mobile clients communicate through the API Gateway / BFF.

I chose the Backend for Frontend (BFF) pattern so that the clients don't need to communicate directly with the internal services.

The BFF provides a client-facing API while hiding the internal architecture.

It also gives me a place to handle things such as authentication, routing, and client-specific requirements.

This means the clients don't need to know whether a request is handled by the Memory Service, Ask Service, or something else.

The internal architecture can evolve without making the clients aware of every change.

2. Each service owns its data

One principle I wanted to follow was:

Don't share databases between services.

A service owns the data that belongs to its responsibility.

In this architecture, the Memory Service owns both the database and vector database.

If another service needs memory data, it doesn't connect directly to either database.

Instead, it calls the Memory Service through an internal API.

For example:

This keeps the data ownership clear and prevents other services from becoming coupled to the Memory Service's storage implementation.

If the storage implementation changes later, the other service shouldn't need to know.

3. Separation of concerns

Each component has a specific responsibility.

The BFF deals with client-facing communication.

The Memory Service deals with memory.

The Ask Service deals with the AI interaction.

The LLM generates the response.

The databases provide persistence.

This separation makes the architecture easier to reason about.

It also means that AI-specific logic doesn't have to spread throughout the rest of the application.

4. DDD and bounded contexts

The separation between Memory Service and Ask Service wasn't just about splitting the code into smaller services.

I was thinking about the different domains and responsibilities involved.

The Memory Service represents the memory domain:

• creating memories

• retrieving memories

• searching memories

• managing memory data

The Ask Service represents the AI interaction:

• receiving a question

• deciding what information is needed

• requesting relevant memories

• constructing context

• calling the LLM

• generating the answer

They are closely related, but they are not the same responsibility.

This is where the idea of bounded contexts from Domain-Driven Design helped me think about the boundaries.

The Ask Service uses the Memory Service, but it doesn't become responsible for managing memory itself.

5. Encapsulation and API contracts

The service boundary also creates an important form of encapsulation.

The Ask Service doesn't need to know how the Memory Service stores or searches memories.

It only needs to know what the Memory Service provides through its API.

For example, the Ask Service might ask for relevant memories, without knowing whether the Memory Service uses PostgreSQL, pgvector, a different vector store, or some other implementation in the future.

The API becomes the contract between the services.

This keeps implementation details inside the service that owns them.

These principles gave me the initial structure of Second-Memory.

It wasn't a complicated architecture, and that was intentional.

I wanted clear boundaries without creating unnecessary complexity.

With the boundaries in place, I could start looking at the individual technical decisions.

I had the idea for Second-Memory.

But at that point, it was still just an idea. It was too vague to start thinking about architecture or implementation.

I wanted to develop the idea further before I started building it.

So I put on the Product Owner hat.

I followed the kind of process I had experienced in my professional work.

I started by defining the initiative. I wrote down what I was trying to achieve and what I wanted the product to be.

Then I wrote a PRD.

Writing it made me think through the product in more detail: the problem, the user, the main use cases, the scope, and what I wanted to achieve with the first version.

From there, I created user stories.

This helped me look at the product from the user's perspective rather than thinking about how I was going to implement it.

I then organised the stories into an initial backlog and broke the work down into tasks that I could eventually implement.

There was another reason I wanted to document all of this.

I knew I would be using AI tools to help me build the project.

Instead of explaining the project from scratch every time I asked an AI coding tool for help, I could give it the product documents as context.

The PRD and user stories became a shared reference point — for me as the developer, and for the AI tools I was working with.

The idea started in my head as something fairly vague.

By the end of this process, I had something much more concrete:

an idea → a product definition → user stories → work I could build.

I wanted to learn how to build AI systems.

So I started looking for a project.

At the time, I was using ChatGPT like everyone else. I would type something, and it would answer.

That's what makes a chatbot useful.

But I started thinking about something different.

Sometimes I don't want an answer.

Sometimes I don't want advice.

I don't want another question.

I just want to write down what I'm thinking.

A thought.

An idea.

Something that happened today.

A feeling.

Something I'm not even sure is important yet.

Just like talking to myself.

What if the AI didn't respond?

Imagine opening an AI application and writing:

"I've been thinking about learning AI more seriously. I don't know where this will take me, but I feel like I should start building something."

And then...

Nothing.

No advice.

No suggestions.

No follow-up question.

It just keeps it.

That's it.

Then a few weeks later, I could ask:

"What have I been thinking about learning AI?"

And now the system responds.

Not based on a generic conversation.

Not based on what I happened to tell it five minutes ago.

But based on the things I had previously written down.

That was the idea behind Second-Memory.

A different kind of AI interaction

Most chatbots are built around a simple interaction:

You → AI → You → AI → You

Every message expects a response.

I wanted something different:

Me → Second-Memory

Me → Second-Memory

Me → Second-Memory

...

And only when I have a question:

Me → Second-Memory → Answer based on what I've written

The system isn't there to constantly talk to me.

It's there to remember.

That's where the name came from:

Second-Memory.

A place where I can put the things I might otherwise forget.

Why build this?

This started as an AI learning project.

I wanted to understand what it actually takes to build an AI-powered system rather than simply calling an LLM API.

And this idea seemed like a good problem to explore.

How do you store someone's thoughts?

How do you find the relevant ones later?

How do you understand that two thoughts written months apart might be related?

How do you give an AI enough context to answer a question without giving it everything?

How do you know whether the answer is actually grounded in what the person wrote?

I didn't know the answers yet.

I just had the idea.

So I created a repository.

The first commit was:

second-memoryYour second memory - capture today, remember forever

And that's where the project began.