Skip to main content

Command Palette

Search for a command to run...

How We Built Infrastructure That Remembers Using Hindsight

Updated
•25 min read•View as Markdown
How We Built Infrastructure That Remembers Using Hindsight

Production incidents rarely happen at a convenient time.

Imagine it is 3:14 AM. A payment API suddenly starts timing out, latency climbs above five seconds, and an engineer on call begins investigating. CPU looks suspicious, so pods are checked. Network latency is investigated. A few common fixes are considered.

Twenty minutes later, the team finally checks the database connection pool.

It is saturated at 99%.

The team increases the pool capacity, latency returns to normal, and the incident is resolved.

But there is a frustrating question:

Hasn't this happened before?

A previous incident had produced almost the same symptoms. The post-mortem contained the root cause, the metrics that should have been checked first, and an important warning: restarting the API could trigger a connection storm and make the situation worse.

The information existed.

The problem was that the system did not remember it when the next incident happened.

That is the problem we wanted to solve with Memora.

Memora — Infrastructure that remembers.

Memora is a memory-powered incident response agent that uses Hindsight to retain operational experience from resolved incidents and recall that experience when similar incidents occur.

Instead of treating every incident as a completely new problem, Memora allows an AI agent to learn from previous investigations, successful fixes, failed approaches, and post-incident lessons.


The Problem: Engineering Teams Keep Rediscovering the Same Knowledge

Modern engineering teams have plenty of observability data.

They have:

  • Logs

  • Metrics

  • Alerts

  • Incident tickets

  • Runbooks

  • Deployment history

  • Post-mortems

  • Slack or Teams discussions

  • Documentation

But this information is usually fragmented across different systems.

When a new incident occurs, an engineer has to reconstruct the context manually.

This creates a hidden operational cost:

Repeated rediscovery.

A previous engineer may already have discovered the root cause. Someone may already have tested an approach that failed. A runbook may already explain the correct investigation path.

Yet the next incident can still begin from zero.

The problem is therefore not simply a lack of data.

Engineering teams need persistent operational memory.


Meet Memora

Memora is an incident response agent designed around a simple idea:

Infrastructure should remember what engineers have already learned.

The workflow is straightforward:

Incident → Investigate → Recall → Compare → Resolve → Learn

When an incident occurs, Memora examines the current evidence and recalls relevant historical experiences from Hindsight.

It then compares the current situation with previous incidents rather than blindly copying an old solution.

Once the incident is resolved, the important operational experience is retained for future investigations.

This creates a learning loop:

                    ┌──────────────────┐
                    │  Production      │
                    │  Incident        │
                    └────────┬─────────┘
                             │
                             ▼
                    ┌──────────────────┐
                    │ Current Evidence │
                    │ Logs / Metrics   │
                    └────────┬─────────┘
                             │
                             ▼
                    ┌──────────────────┐
                    │ Memora Agent     │
                    │ Investigation    │
                    └────────┬─────────┘
                             │
                   ┌─────────┴─────────┐
                   ▼                   ▼
          ┌────────────────┐   ┌────────────────┐
          │ Current        │   │ Hindsight      │
          │ Evidence       │   │ Memory         │
          └────────┬───────┘   └───────┬────────┘
                   │                   │
                   └─────────┬─────────┘
                             ▼
                    ┌──────────────────┐
                    │ Compare & Reason │
                    └────────┬─────────┘
                             │
                             ▼
                    ┌──────────────────┐
                    │ Investigation    │
                    │ & Resolution     │
                    └────────┬─────────┘
                             │
                             ▼
                    ┌──────────────────┐
                    │ Retain Learning  │
                    │ in Hindsight     │
                    └──────────────────┘

The System Architecture

Memora separates application state from reusable operational experience.

┌──────────────────────────────────────────────────────┐
│                  React + Vite UI                     │
│     Incident Workspace | Evidence | Agent | Memory  │
└─────────────────────────┬────────────────────────────┘
                          │
                       HTTP/REST
                          │
┌─────────────────────────▼────────────────────────────┐
│                 Node.js / Express API                │
├─────────────────────────┬────────────────────────────┤
│      Supabase           │       Hindsight            │
│      PostgreSQL         │       Memory Engine        │
│                         │                            │
│ • Incidents             │ • Historical experiences   │
│ • Evidence              │ • Successful approaches   │
│ • Investigation steps   │ • Failed approaches       │
│ • Runbooks              │ • Lessons learned         │
│ • Resolutions           │ • Operational context     │
└─────────────────────────┴────────────────────────────┘

Supabase PostgreSQL

Supabase stores the application's structured state:

  • Incidents

  • Incident evidence

  • Investigation steps

  • Runbooks

  • Resolutions

  • Audit information

It acts as the application's source of truth.

Hindsight

Hindsight is used for a different purpose: reusable agent experience.

Instead of storing the entire history of raw logs, Memora retains distilled operational knowledge that can be useful during future investigations.

This separation is important.

PostgreSQL answers:

What happened in the application?

Hindsight helps answer:

What have we learned from similar incidents before?


Why We Didn't Just Store Raw Logs

One of the first design questions was:

Why not simply put every incident log into a vector database and search for similar logs?

Because raw logs are noisy.

A typical production log can contain:

  • timestamps

  • container IDs

  • request IDs

  • stack traces

  • repeated messages

  • transient values

  • infrastructure-specific metadata

Finding a similar string does not necessarily mean finding a useful operational experience.

An alert saying:

Payment API latency exceeded 5 seconds

does not tell an engineer:

  • what caused it

  • which checks were performed

  • what failed

  • what worked

  • what should be avoided next time

  • what evidence confirmed the root cause

Memora therefore focuses on experience rather than raw history.


What Memora Remembers

After an incident is resolved, Memora extracts the useful operational knowledge into a structured experience.

For example:

const experienceToRetain = {
  type: "incident_experience",

  service: "payment-api",

  incident_key: "INC-1001",

  symptoms: [
    "Payment API latency",
    "Increasing timeouts"
  ],

  important_evidence: [
    "DB connections: 99%",
    "API latency: 5.2s"
  ],

  root_cause:
    "Database connection pool exhaustion",

  investigation_path: [
    "Checked CPU and ruled out throttling",
    "Inspected DB connection pool",
    "Confirmed connection exhaustion"
  ],

  failed_approaches: [
    "Restarting the API caused a cold-start connection storm"
  ],

  successful_approaches: [
    "Increased database connection pool capacity"
  ],

  runbook: "DB-CONNECTION-POOL-03",

  outcome:
    "Latency returned to normal",

  lesson:
    "When payment-api latency coincides with high DB connections, inspect the connection pool before restarting."
};

The important part is not simply remembering the final fix.

Memora also remembers how the team got there.

That includes failed approaches.


Why Failed Approaches Matter

During an incident, knowing what not to do can be just as valuable as knowing what worked.

Suppose a previous incident established that restarting the API caused a connection storm.

If the same symptoms appear again, that information should be available to the engineer before they repeat the same action.

Therefore Memora retains both:

Successful approach
        +
Failed approach
        +
Evidence
        +
Root cause
        +
Lesson

This turns historical incidents into reusable operational experience.


The Most Important Design Principle: Memory Is Not Truth

Adding memory to an AI agent introduces another problem.

What happens when historical experience looks relevant, but the current incident is actually different?

This is where Memora deliberately avoids blindly trusting memory.

We created explicit evidence labels:

OBSERVED

Facts directly supported by current telemetry or logs.

HISTORICAL

Information recalled from previous incidents through Hindsight.

HYPOTHESIS

A possible explanation that still needs to be tested.

CONFIRMED

A hypothesis supported by current evidence.

RULED_OUT

A hypothesis that was investigated and rejected.

This distinction allows the agent to reason about history without confusing historical knowledge with current facts.


Current Evidence Always Wins

Consider a second incident.

The symptoms look familiar:

Payment API latency
Increasing timeouts

Memora recalls the previous database connection pool incident.

But this time the evidence is different:

DB connection utilization: 34%
Redis memory utilization: 96%

A naive memory system might say:

"This looks like the previous incident. Increase the database connection pool."

Memora should not do that.

Instead, it recognizes the similarity and the difference.

Conceptually:

if (highRedisMemory && !highDbConnections && previousDbPoolIncident) {

  differences.push(
    "Current DB connection utilization is normal."
  );

  differences.push(
    "Redis memory pressure is significantly higher."
  );

  hypotheses.push({
    title: "Redis memory pressure",
    label: "HYPOTHESIS"
  });

  hypotheses.push({
    title: "Database connection pool exhaustion",
    label: "RULED_OUT"
  });
}

The historical incident is still useful.

But it is treated as evidence, not truth.

This is one of the most important principles behind Memora:

History should inform investigation, not override current reality.


The Three-Scenario Demo

The easiest way to demonstrate Memora is through three incidents.

Scenario 1 — Teach Memora

INC-1001

Service:

payment-api

Symptoms:

Latency: 5.2s
Timeouts increasing
DB connections: 99%

Memora investigates the incident.

The engineer confirms:

Root cause:
Database connection pool exhaustion

The resolution is recorded.

Memora then retains the experience in Hindsight.

At this point, Memora has learned something useful.


Scenario 2 — Memory Changes the Investigation

INC-1007

Several weeks later, another payment API incident occurs.

The current evidence is:

Latency: 5.1s
Timeouts increasing
DB connections: 97%

Memora recalls INC-1001.

Instead of beginning from a generic checklist, it can surface:

Historical incident:
INC-1001

Similar symptoms:
Payment API latency + high DB connections

Historical root cause:
Connection pool exhaustion

Previous successful approach:
Increase connection pool capacity

Previous failed approach:
Restarting the API caused a connection storm

Recommended next check:
Inspect DB connection pool utilization

The key point is not that Memora automatically fixes the incident.

The key point is:

The previous incident changes the investigation starting point.


Scenario 3 — Memora Knows When History Doesn't Apply

INC-1008

Another payment API incident occurs.

The symptoms initially look similar:

High API latency
Increasing timeouts

Memora again recalls INC-1001.

But the current evidence says:

DB connections: 34%
Redis memory: 96%

Memora identifies the divergence.

Instead of recommending the historical database fix, it can explain:

Historical match:
INC-1001

Why it is relevant:
Similar payment API latency

Why it may not apply:
Current DB connection utilization is normal.

Current evidence:
Redis memory pressure is high.

Historical DB-pool hypothesis:
RULED_OUT

The investigation can then move toward the Redis-related hypothesis.

This third scenario is critical because it demonstrates that Memora is not simply a retrieval system.

It uses memory while remaining grounded in current evidence.


What Makes Hindsight Central to Memora?

The core value of Memora depends on the memory loop.

Without memory:

Incident
   ↓
Investigation
   ↓
Resolution
   ↓
End

With Hindsight:

Incident
   ↓
Investigation
   ↓
Resolution
   ↓
Retain experience
   ↓
Hindsight
   ↓
Future incident
   ↓
Recall relevant experience
   ↓
Better-informed investigation

The system therefore becomes capable of accumulating operational knowledge over time.

This aligns with the idea behind memory-powered agents: an agent should not behave as if every interaction is its first experience.


Making Memory Visible

We also wanted the memory layer to be visible rather than hidden behind the agent.

Inside the Memora incident workspace, historical memory can be surfaced alongside the current investigation.

An engineer should be able to see:

CURRENT EVIDENCE
DB connections: 97%
Latency: 5.1s

        ↓

HISTORICAL MEMORY
INC-1001
DB pool exhaustion

        ↓

WHY IT MATTERS
Similar service + similar latency +
high DB connection utilization

        ↓

PREVIOUSLY SUCCESSFUL
Increase connection pool capacity

        ↓

PREVIOUSLY FAILED
Restarting API

        ↓

NEXT CHECK
Inspect DB connection pool

This makes the agent's reasoning auditable.

Instead of:

"Trust me. Do this."

The system communicates:

"This historical experience is relevant because these current signals match. Here is what happened before, and here is what is different now."

That distinction is important for engineering workflows where operators need to understand and verify recommendations.


4 Lessons We Learned Building Memora

1. Store Experiences, Not Just Data

Raw logs are valuable for observability, but they are not necessarily useful as long-term agent memory.

The valuable memory is the distilled experience:

Symptoms
+
Evidence
+
Investigation
+
Failed approaches
+
Successful approaches
+
Root cause
+
Lesson

2. Remember Failures Too

An incident response agent should not only remember:

"This fixed the problem."

It should also remember:

"We tried this and it didn't work."

Avoiding a known dead end can save valuable investigation time.


3. Historical Memory Must Remain Grounded

A previous incident is evidence, not proof that the current incident has the same cause.

Memora therefore compares historical experiences against current telemetry.

The system should be willing to say:

"This looks similar, but the current evidence does not support the same root cause."

That behavior is more important than simply retrieving a highly similar historical incident.


4. Explain Why Memory Influenced the Agent

Memory becomes much more useful when the engineer can understand why it was retrieved.

For example:

Influenced by INC-1001

Because:
• Same service
• Similar latency pattern
• DB connections currently at 97%
• Previous incident involved DB pool exhaustion

This makes the relationship between memory and the current investigation explicit.


An Honest Limitation

Memora is not intended to replace observability platforms, incident-management systems, or experienced SREs.

It also does not automatically execute destructive production actions.

The current system focuses on a narrower problem:

Helping engineers reuse operational knowledge during incident investigation.

Another important limitation is that memory quality depends on the quality of the experiences retained.

If an incident is incorrectly diagnosed and that incorrect conclusion is retained, future investigations could be influenced by it.

That is why Memora emphasizes:

  • verified resolutions

  • evidence-backed conclusions

  • explicit uncertainty

  • current telemetry

  • visible historical context

  • human verification

Memory should make an agent more informed—not make it blindly confident.


Where Memora Goes Next

The current implementation focuses on one workflow:

Incident investigation.

The same memory architecture could eventually support broader engineering workflows:

  • Deployment troubleshooting

  • CI/CD failure analysis

  • Code review knowledge

  • Customer support history

  • Infrastructure changes

  • Runbook evolution

  • Post-mortem analysis

The long-term idea is larger than incident response.

Engineering organizations continuously generate knowledge.

The challenge is making that knowledge reusable when it matters.


Final Takeaway

The most interesting part of building an AI agent is not making it answer a question.

It is making the agent better because it has experienced something before.

Memora explores that idea through incident response.

A production incident teaches the system.

A future incident recalls that experience.

The current evidence determines whether the memory still applies.

And the resulting investigation becomes another experience that can be remembered.

That creates a simple but powerful loop:

Experience → Memory → Recall → Reason → Learn

We built Memora around one belief:

Infrastructure should not have to forget what engineers already learned.

Resources

  • Hindsight GitHub Repository

  • Hinds

    How We Built Infrastructure That Remembers Using Hindsight

    Production incidents rarely happen at a convenient time.

    Imagine it is 3:14 AM. A payment API suddenly starts timing out, latency climbs above five seconds, and an engineer on call begins investigating. CPU looks suspicious, so pods are checked. Network latency is investigated. A few common fixes are considered.

    Twenty minutes later, the team finally checks the database connection pool.

    It is saturated at 99%.

    The team increases the pool capacity, latency returns to normal, and the incident is resolved.

    But there is a frustrating question:

    Hasn't this happened before?

    A previous incident had produced almost the same symptoms. The post-mortem contained the root cause, the metrics that should have been checked first, and an important warning: restarting the API could trigger a connection storm and make the situation worse.

    The information existed.

    The problem was that the system did not remember it when the next incident happened.

    That is the problem we wanted to solve with Memora.

    Memora — Infrastructure that remembers.

    Memora is a memory-powered incident response agent that uses Hindsight to retain operational experience from resolved incidents and recall that experience when similar incidents occur.

    Instead of treating every incident as a completely new problem, Memora allows an AI agent to learn from previous investigations, successful fixes, failed approaches, and post-incident lessons.


    The Problem: Engineering Teams Keep Rediscovering the Same Knowledge

    Modern engineering teams have plenty of observability data.

    They have:

    • Logs

    • Metrics

    • Alerts

    • Incident tickets

    • Runbooks

    • Deployment history

    • Post-mortems

    • Slack or Teams discussions

    • Documentation

    But this information is usually fragmented across different systems.

    When a new incident occurs, an engineer has to reconstruct the context manually.

    This creates a hidden operational cost:

    Repeated rediscovery.

    A previous engineer may already have discovered the root cause. Someone may already have tested an approach that failed. A runbook may already explain the correct investigation path.

    Yet the next incident can still begin from zero.

    The problem is therefore not simply a lack of data.

    Engineering teams need persistent operational memory.


    Meet Memora

    Memora is an incident response agent designed around a simple idea:

    Infrastructure should remember what engineers have already learned.

    The workflow is straightforward:

    Incident → Investigate → Recall → Compare → Resolve → Learn

    When an incident occurs, Memora examines the current evidence and recalls relevant historical experiences from Hindsight.

    It then compares the current situation with previous incidents rather than blindly copying an old solution.

    Once the incident is resolved, the important operational experience is retained for future investigations.

    This creates a learning loop:

                        ┌──────────────────┐
                        │  Production      │
                        │  Incident        │
                        └────────┬─────────┘
                                 │
                                 ▼
                        ┌──────────────────┐
                        │ Current Evidence │
                        │ Logs / Metrics   │
                        └────────┬─────────┘
                                 │
                                 ▼
                        ┌──────────────────┐
                        │ Memora Agent     │
                        │ Investigation    │
                        └────────┬─────────┘
                                 │
                       ┌─────────┴─────────┐
                       ▼                   ▼
              ┌────────────────┐   ┌────────────────┐
              │ Current        │   │ Hindsight      │
              │ Evidence       │   │ Memory         │
              └────────┬───────┘   └───────┬────────┘
                       │                   │
                       └─────────┬─────────┘
                                 ▼
                        ┌──────────────────┐
                        │ Compare & Reason │
                        └────────┬─────────┘
                                 │
                                 ▼
                        ┌──────────────────┐
                        │ Investigation    │
                        │ & Resolution     │
                        └────────┬─────────┘
                                 │
                                 ▼
                        ┌──────────────────┐
                        │ Retain Learning  │
                        │ in Hindsight     │
                        └──────────────────┘
    

    The System Architecture

    Memora separates application state from reusable operational experience.

    ┌──────────────────────────────────────────────────────┐
    │                  React + Vite UI                     │
    │     Incident Workspace | Evidence | Agent | Memory  │
    └─────────────────────────┬────────────────────────────┘
                              │
                           HTTP/REST
                              │
    ┌─────────────────────────▼────────────────────────────┐
    │                 Node.js / Express API                │
    ├─────────────────────────┬────────────────────────────┤
    │      Supabase           │       Hindsight            │
    │      PostgreSQL         │       Memory Engine        │
    │                         │                            │
    │ • Incidents             │ • Historical experiences   │
    │ • Evidence              │ • Successful approaches   │
    │ • Investigation steps   │ • Failed approaches       │
    │ • Runbooks              │ • Lessons learned         │
    │ • Resolutions           │ • Operational context     │
    └─────────────────────────┴────────────────────────────┘
    

    Supabase PostgreSQL

    Supabase stores the application's structured state:

    • Incidents

    • Incident evidence

    • Investigation steps

    • Runbooks

    • Resolutions

    • Audit information

    It acts as the application's source of truth.

    Hindsight

    Hindsight is used for a different purpose: reusable agent experience.

    Instead of storing the entire history of raw logs, Memora retains distilled operational knowledge that can be useful during future investigations.

    This separation is important.

    PostgreSQL answers:

    What happened in the application?

    Hindsight helps answer:

    What have we learned from similar incidents before?


    Why We Didn't Just Store Raw Logs

    One of the first design questions was:

    Why not simply put every incident log into a vector database and search for similar logs?

    Because raw logs are noisy.

    A typical production log can contain:

    • timestamps

    • container IDs

    • request IDs

    • stack traces

    • repeated messages

    • transient values

    • infrastructure-specific metadata

    Finding a similar string does not necessarily mean finding a useful operational experience.

    An alert saying:

    Payment API latency exceeded 5 seconds
    

    does not tell an engineer:

    • what caused it

    • which checks were performed

    • what failed

    • what worked

    • what should be avoided next time

    • what evidence confirmed the root cause

    Memora therefore focuses on experience rather than raw history.


    What Memora Remembers

    After an incident is resolved, Memora extracts the useful operational knowledge into a structured experience.

    For example:

    const experienceToRetain = {
      type: "incident_experience",
    
      service: "payment-api",
    
      incident_key: "INC-1001",
    
      symptoms: [
        "Payment API latency",
        "Increasing timeouts"
      ],
    
      important_evidence: [
        "DB connections: 99%",
        "API latency: 5.2s"
      ],
    
      root_cause:
        "Database connection pool exhaustion",
    
      investigation_path: [
        "Checked CPU and ruled out throttling",
        "Inspected DB connection pool",
        "Confirmed connection exhaustion"
      ],
    
      failed_approaches: [
        "Restarting the API caused a cold-start connection storm"
      ],
    
      successful_approaches: [
        "Increased database connection pool capacity"
      ],
    
      runbook: "DB-CONNECTION-POOL-03",
    
      outcome:
        "Latency returned to normal",
    
      lesson:
        "When payment-api latency coincides with high DB connections, inspect the connection pool before restarting."
    };
    

    The important part is not simply remembering the final fix.

    Memora also remembers how the team got there.

    That includes failed approaches.


    Why Failed Approaches Matter

    During an incident, knowing what not to do can be just as valuable as knowing what worked.

    Suppose a previous incident established that restarting the API caused a connection storm.

    If the same symptoms appear again, that information should be available to the engineer before they repeat the same action.

    Therefore Memora retains both:

    Successful approach
            +
    Failed approach
            +
    Evidence
            +
    Root cause
            +
    Lesson
    

    This turns historical incidents into reusable operational experience.


    The Most Important Design Principle: Memory Is Not Truth

    Adding memory to an AI agent introduces another problem.

    What happens when historical experience looks relevant, but the current incident is actually different?

    This is where Memora deliberately avoids blindly trusting memory.

    We created explicit evidence labels:

    OBSERVED

    Facts directly supported by current telemetry or logs.

    HISTORICAL

    Information recalled from previous incidents through Hindsight.

    HYPOTHESIS

    A possible explanation that still needs to be tested.

    CONFIRMED

    A hypothesis supported by current evidence.

    RULED_OUT

    A hypothesis that was investigated and rejected.

    This distinction allows the agent to reason about history without confusing historical knowledge with current facts.


    Current Evidence Always Wins

    Consider a second incident.

    The symptoms look familiar:

    Payment API latency
    Increasing timeouts
    

    Memora recalls the previous database connection pool incident.

    But this time the evidence is different:

    DB connection utilization: 34%
    Redis memory utilization: 96%
    

    A naive memory system might say:

    "This looks like the previous incident. Increase the database connection pool."

    Memora should not do that.

    Instead, it recognizes the similarity and the difference.

    Conceptually:

    if (highRedisMemory && !highDbConnections && previousDbPoolIncident) {
    
      differences.push(
        "Current DB connection utilization is normal."
      );
    
      differences.push(
        "Redis memory pressure is significantly higher."
      );
    
      hypotheses.push({
        title: "Redis memory pressure",
        label: "HYPOTHESIS"
      });
    
      hypotheses.push({
        title: "Database connection pool exhaustion",
        label: "RULED_OUT"
      });
    }
    

    The historical incident is still useful.

    But it is treated as evidence, not truth.

    This is one of the most important principles behind Memora:

    History should inform investigation, not override current reality.


    The Three-Scenario Demo

    The easiest way to demonstrate Memora is through three incidents.

    Scenario 1 — Teach Memora

    INC-1001

    Service:

    payment-api
    

    Symptoms:

    Latency: 5.2s
    Timeouts increasing
    DB connections: 99%
    

    Memora investigates the incident.

    The engineer confirms:

    Root cause:
    Database connection pool exhaustion
    

    The resolution is recorded.

    Memora then retains the experience in Hindsight.

    At this point, Memora has learned something useful.


    Scenario 2 — Memory Changes the Investigation

    INC-1007

    Several weeks later, another payment API incident occurs.

    The current evidence is:

    Latency: 5.1s
    Timeouts increasing
    DB connections: 97%
    

    Memora recalls INC-1001.

    Instead of beginning from a generic checklist, it can surface:

    Historical incident:
    INC-1001
    
    Similar symptoms:
    Payment API latency + high DB connections
    
    Historical root cause:
    Connection pool exhaustion
    
    Previous successful approach:
    Increase connection pool capacity
    
    Previous failed approach:
    Restarting the API caused a connection storm
    
    Recommended next check:
    Inspect DB connection pool utilization
    

    The key point is not that Memora automatically fixes the incident.

    The key point is:

    The previous incident changes the investigation starting point.


    Scenario 3 — Memora Knows When History Doesn't Apply

    INC-1008

    Another payment API incident occurs.

    The symptoms initially look similar:

    High API latency
    Increasing timeouts
    

    Memora again recalls INC-1001.

    But the current evidence says:

    DB connections: 34%
    Redis memory: 96%
    

    Memora identifies the divergence.

    Instead of recommending the historical database fix, it can explain:

    Historical match:
    INC-1001
    
    Why it is relevant:
    Similar payment API latency
    
    Why it may not apply:
    Current DB connection utilization is normal.
    
    Current evidence:
    Redis memory pressure is high.
    
    Historical DB-pool hypothesis:
    RULED_OUT
    

    The investigation can then move toward the Redis-related hypothesis.

    This third scenario is critical because it demonstrates that Memora is not simply a retrieval system.

    It uses memory while remaining grounded in current evidence.


    What Makes Hindsight Central to Memora?

    The core value of Memora depends on the memory loop.

    Without memory:

    Incident
       ↓
    Investigation
       ↓
    Resolution
       ↓
    End
    

    With Hindsight:

    Incident
       ↓
    Investigation
       ↓
    Resolution
       ↓
    Retain experience
       ↓
    Hindsight
       ↓
    Future incident
       ↓
    Recall relevant experience
       ↓
    Better-informed investigation
    

    The system therefore becomes capable of accumulating operational knowledge over time.

    This aligns with the idea behind memory-powered agents: an agent should not behave as if every interaction is its first experience.


    Making Memory Visible

    We also wanted the memory layer to be visible rather than hidden behind the agent.

    Inside the Memora incident workspace, historical memory can be surfaced alongside the current investigation.

    An engineer should be able to see:

    CURRENT EVIDENCE
    DB connections: 97%
    Latency: 5.1s
    
            ↓
    
    HISTORICAL MEMORY
    INC-1001
    DB pool exhaustion
    
            ↓
    
    WHY IT MATTERS
    Similar service + similar latency +
    high DB connection utilization
    
            ↓
    
    PREVIOUSLY SUCCESSFUL
    Increase connection pool capacity
    
            ↓
    
    PREVIOUSLY FAILED
    Restarting API
    
            ↓
    
    NEXT CHECK
    Inspect DB connection pool
    

    This makes the agent's reasoning auditable.

    Instead of:

    "Trust me. Do this."

    The system communicates:

    "This historical experience is relevant because these current signals match. Here is what happened before, and here is what is different now."

    That distinction is important for engineering workflows where operators need to understand and verify recommendations.


    4 Lessons We Learned Building Memora

    1. Store Experiences, Not Just Data

    Raw logs are valuable for observability, but they are not necessarily useful as long-term agent memory.

    The valuable memory is the distilled experience:

    Symptoms
    +
    Evidence
    +
    Investigation
    +
    Failed approaches
    +
    Successful approaches
    +
    Root cause
    +
    Lesson
    

    2. Remember Failures Too

    An incident response agent should not only remember:

    "This fixed the problem."

    It should also remember:

    "We tried this and it didn't work."

    Avoiding a known dead end can save valuable investigation time.


    3. Historical Memory Must Remain Grounded

    A previous incident is evidence, not proof that the current incident has the same cause.

    Memora therefore compares historical experiences against current telemetry.

    The system should be willing to say:

    "This looks similar, but the current evidence does not support the same root cause."

    That behavior is more important than simply retrieving a highly similar historical incident.


    4. Explain Why Memory Influenced the Agent

    Memory becomes much more useful when the engineer can understand why it was retrieved.

    For example:

    Influenced by INC-1001
    
    Because:
    • Same service
    • Similar latency pattern
    • DB connections currently at 97%
    • Previous incident involved DB pool exhaustion
    

    This makes the relationship between memory and the current investigation explicit.


    An Honest Limitation

    Memora is not intended to replace observability platforms, incident-management systems, or experienced SREs.

    It also does not automatically execute destructive production actions.

    The current system focuses on a narrower problem:

    Helping engineers reuse operational knowledge during incident investigation.

    Another important limitation is that memory quality depends on the quality of the experiences retained.

    If an incident is incorrectly diagnosed and that incorrect conclusion is retained, future investigations could be influenced by it.

    That is why Memora emphasizes:

    • verified resolutions

    • evidence-backed conclusions

    • explicit uncertainty

    • current telemetry

    • visible historical context

    • human verification

    Memory should make an agent more informed—not make it blindly confident.


    Where Memora Goes Next

    The current implementation focuses on one workflow:

    Incident investigation.

    The same memory architecture could eventually support broader engineering workflows:

    • Deployment troubleshooting

    • CI/CD failure analysis

    • Code review knowledge

    • Customer support history

    • Infrastructure changes

    • Runbook evolution

    • Post-mortem analysis

    The long-term idea is larger than incident response.

    Engineering organizations continuously generate knowledge.

    The challenge is making that knowledge reusable when it matters.


    Final Takeaway

    The most interesting part of building an AI agent is not making it answer a question.

    It is making the agent better because it has experienced something before.

    Memora explores that idea through incident response.

    A production incident teaches the system.

    A future incident recalls that experience.

    The current evidence determines whether the memory still applies.

    And the resulting investigation becomes another experience that can be remembered.

    That creates a simple but powerful loop:

    Experience → Memory → Recall → Reason → Learn

    We built Memora around one belief:

    Infrastructure should not have to forget what engineers already learned.

    Resources

    Resources

6 views