DEV Community: Vikash Kumar

How Does a Chatbot Remember? The Engineering Behind Context and Memory

Vikash Kumar — Tue, 27 Oct 2026 03:46:33 +0000

One of the interesting things about using a chatbot is how naturally a conversation can continue.

I can ask a question, provide additional information, refer to something mentioned earlier, and expect the assistant to understand what I mean.

But while studying how chatbots work, I learned an important distinction:

A continuous conversation is something the application must create. It is not automatically provided by every individual LLM call.

What happens when I send a follow-up message?

Imagine this conversation:

User: My food hasn't arrived.

Assistant: Would you like to wait or request a refund?

User: I want a refund.

The last message makes sense because of the earlier exchange.

But what happens if the application sends only “I want a refund” to the model?

The model may not have the information needed to understand the situation. The application needs to preserve the relevant conversation history and include it in the next request.

This was my first important takeaway: the application is responsible for maintaining the conversation.

Memory is not necessarily inside the model

Suppose I tell a chatbot that my preferred programming language is Java.

Later, I ask which language I prefer.

It might seem that the model has permanently learned something about me.

However, the application could have stored that preference, retrieved it when needed, and supplied it as context for the new request.

The process could look like this:

User Information
       ↓
Application Memory
       ↓
Retrieve Relevant Information
       ↓
Add to Current Context
       ↓
LLM Generates Response

This is different from training the model and updating its learned parameters.

That distinction changed how I think about chatbot memory. The user experience may look like persistent memory even when the application is retrieving stored information and passing it to the model.

More context is not always better

Another concept that stood out to me was the context window.

A model can process only a limited amount of information in a given interaction.

An application might store hundreds or thousands of messages, but it does not necessarily need to send every message with every request.

It can select recent messages, summarize older exchanges, or retrieve only information relevant to the current question.

Why does this matter?

Because repeatedly including a growing conversation can increase input token usage, which can affect cost and latency. It also consumes some of the model's available context capacity.

The important design question is not simply:

“How much context can I send?”

It is:

“Which information does the model actually need to answer this request well?”

Streaming is a user-experience decision

The lecture also connected next-token generation to streaming.

Instead of waiting for an entire answer before showing anything, an application can display generated content progressively as it arrives.

This can make an application feel more responsive, although it does not necessarily make the underlying model generate the complete response faster.

It is another reminder that the experience users see depends on application design, not just model capabilities.

And then there is hallucination

A model can generate an answer that sounds convincing without the answer being factually correct.

Fluent language and verified information are two different things.

For a real application, this matters enormously.

Imagine a support chatbot answering questions about an order. It should retrieve the actual order status rather than inventing one. If an action requires authorization, the application must enforce that authorization instead of trusting a prompt alone.

Depending on the use case, retrieval, external APIs, tool calling, validation, guardrails, and evaluation can improve reliability.

They reduce certain risks, but they do not guarantee that every answer will be correct.

What I'm taking away as a software engineer

I'm beginning to see chatbot engineering as more than sending prompts to a model.

The application needs to manage conversation history, select context, retrieve relevant information, control access to tools, and handle responses appropriately.

These are familiar software engineering concerns applied to a new kind of capability.

My biggest takeaway from this lecture is:

A good chatbot does not merely generate responses. Its application must provide the right context and manage those responses responsibly.

As I continue studying Forward Deployed Engineering and Generative AI, I want to understand not only how models work, but also how to build reliable applications around them.

An LLM Is Not an Application: What I’m Learning About Building GenAI with Backend Engineering

Vikash Kumar — Fri, 23 Oct 2026 11:55:38 +0000

As I continue learning Forward Deployed Engineering and Generative AI, I am beginning to look at AI from a software engineer's perspective.

My third lecture focused on a distinction that seems simple at first:

An LLM is not the same thing as a GenAI application.

An LLM provides language capabilities. But a real application needs software around the model to make those capabilities useful.

The model is only one part of the system

Imagine a customer support team wants to summarize incoming support tickets.

The LLM can generate a summary. But someone still needs to build the system that receives the ticket, sends the right information to the model, handles the response, and makes that result available to the support team.

The overall workflow might look like this:

Support Ticket
      ↓
Backend Application
      ↓
LLM Provider API
      ↓
Generated Summary
      ↓
Backend Application
      ↓
Support Team

The model is important, but it is only one component in the workflow.

What if the model needs information it doesn't have?

Consider a user asking:

“Where is my order right now?”

A model cannot automatically know the user's latest order status. The application needs to retrieve that information from the relevant business system.

The application may need to:

Identify the requested information.
Retrieve the appropriate order data.
Provide the relevant context to the model.
Generate a clear response.
Return the result to the user.

This changes how I think about building AI features.

Instead of asking only, “Which model should I use?”, I also need to ask, “What information does this workflow require, and how will my application obtain it?”

Connecting GenAI to backend engineering

As a Java backend developer, I already understand APIs, services, controllers, external integrations, and data flow.

Those concepts still matter when integrating an LLM.

For example, a Spring Boot application can expose a REST endpoint, accept a support ticket, pass it to a service, and use Spring AI's ChatClient to communicate with a supported model provider.

The model interaction becomes part of an ordinary application workflow.

That does not mean the integration is automatically production-ready. Engineers still need to consider validation, errors, credentials, latency, token usage, and observability.

Why abstractions matter

It is possible to call an LLM provider directly through HTTP. A framework such as Spring AI provides another way to organize those interactions.

As requirements grow to include streaming, structured outputs, tool calling, retrieval, or multiple providers, an abstraction may help reduce provider-specific code in application logic.

The right choice depends on the application. Adding a framework without a real need can create unnecessary complexity, just as it can in other areas of software engineering.

One more design question

My lecture also raised an interesting question: should application instructions and user-provided data be treated as the same thing?

For example:

Instruction: “Summarize this support ticket in two sentences.”
Data: “Payment was deducted, but my order is still processing.”

These serve different purposes. Keeping them conceptually separate is a useful starting point when designing model interactions.

What I'm taking away

My understanding of GenAI is moving from the model itself toward the system built around it.

The model generates language. The application supplies context, connects external capabilities, handles the result, and integrates the model into a real workflow.

This is where my backend engineering experience becomes relevant to the AI systems I want to build.

An LLM can be powerful, but engineering the surrounding application is what turns that capability into a useful feature.

I'm continuing to explore the connection between backend engineering, system design, and production AI applications as part of my Forward Deployed Engineering learning journey.

I Finally Have a Better Mental Model of How an LLM Generates Text

Vikash Kumar — Wed, 21 Oct 2026 16:07:26 +0000

I'm continuing my Forward Deployed Engineering and Generative AI learning journey.

In my previous lecture, I learned an important engineering lesson:

Don't start with the technology. Start with the problem.

This lecture moved one level deeper.

I wanted to understand what actually happens inside an LLM when I type a prompt and it starts generating a response.

Before studying this, it was easy to think about an LLM like this:

Prompt
   ↓
LLM
   ↓
Answer

That's useful at the application level, but it hides almost everything happening inside the model.

The mental model I have now is very different.

It starts with a surprisingly simple idea

At the generation level, an autoregressive LLM repeatedly predicts the next token.

Suppose we provide:

The capital of India is

The model doesn't need to magically generate the complete sentence in one step.

Instead, it estimates which token should come next.

Conceptually:

The capital of India is
        ↓
      Delhi

Then the new sequence becomes:

The capital of India is Delhi

The model predicts again.

And again.

So the basic generation loop is:

Context
  ↓
Predict next token
  ↓
Select token
  ↓
Append token
  ↓
Predict again

That simple idea helped me understand many of the other concepts in the lecture.

But what exactly is a token?

This was another important correction to my mental model.

A token isn't necessarily a complete word.

Depending on the tokenizer, a token can represent:

A complete word
Part of a word
Punctuation
A symbol
Another frequently occurring text fragment

So:

Token ≠ Word

This matters because LLMs operate on tokens rather than directly operating on human concepts like "sentences" or "paragraphs."

Text has to become numbers

Neural networks operate mathematically.

So the text needs to be converted into numerical representations.

Conceptually:

Text
 ↓
Tokens
 ↓
Token IDs
 ↓
Embeddings

Token IDs identify tokens.

But an ID by itself doesn't represent meaning.

The model therefore uses embeddings: learned numerical vectors that can be processed by the neural network.

This distinction was useful:

Token ID
→ Which token is this?

Embedding
→ Numerical representation used for processing

Then I ran into a problem: order matters

Consider:

Dog bites man.

and:

Man bites dog.

Almost the same words.

Completely different meaning.

So the model needs information about position.

It needs to understand not only:

What token is this?

but also:

Where does this token occur?

That gives us the idea of position information.

Then context makes everything more interesting

Consider the word:

bank

In:

I deposited money at the bank.

it has one meaning.

In:

We sat on the bank of the river.

it has another.

The token itself hasn't changed.

The context has.

This leads to an important idea:

A token shouldn't always be interpreted independently of the other tokens around it.

And this is where attention becomes important.

Attention: which other tokens matter?

Consider:

The animal didn't cross the street because it was too tired.

To interpret it, other tokens in the sentence matter.

Attention provides a mechanism for determining which other tokens should influence the current token's representation.

At a high level, I think of it as:

Current token
      ↓
Which other tokens are relevant?
      ↓
Gather useful information
      ↓
Update representation

The lecture introduces Query, Key and Value as the three learned representations used in this mechanism.

The search analogy helped me:

Query
→ What am I looking for?

Key
→ What kind of information do I contain?

Value
→ What information can I provide?

They aren't literally English questions inside the model. They are mathematical vectors.

But the analogy helped me understand their roles.

Attention isn't the Transformer

Another misconception I had to correct:

Transformer = Attention

Not quite.

Attention is an important component of a Transformer.

A Transformer contains repeated processing blocks.

Conceptually:

Input
 ↓
Transformer Block
 ↓
Transformer Block
 ↓
Transformer Block
 ↓
...
 ↓
Final Representation

Each layer can progressively refine the representations.

So instead of imagining the model as one giant operation, I can think about it as repeated transformations of the information.

Finally: how does the model choose the next token?

After the Transformer processes the context, the model produces scores for possible next tokens.

These are called logits.

Then Softmax converts those scores into probabilities.

So:

Context
 ↓
Transformer
 ↓
Logits
 ↓
Softmax
 ↓
Probability distribution

For example:

Delhi      92%
Kolkata     2%
Punjab      2%
Mumbai      1%
...

Now we have another question:

Which token should actually be selected?

That's where decoding comes in.

The model and the decoder are different things

This distinction was particularly useful to me.

The model produces:

Probability distribution

The decoding strategy decides:

Which token gets selected?

For example, greedy decoding chooses the highest-probability token.

Sampling can choose among possible tokens according to their probabilities.

Temperature changes how sharp or flat the distribution becomes before sampling.

So the pipeline becomes:

MODEL
 ↓
Probability Distribution
 ↓
DECODING STRATEGY
 ↓
Selected Token

And then the whole process starts again

This is probably the most important mental model I took from the lecture.

The model selects one token.

That token is appended to the sequence.

The updated sequence becomes the context for the next prediction.

Then the process repeats.

Prompt
 ↓
Predict
 ↓
Select
 ↓
Append
 ↓
Predict
 ↓
Select
 ↓
Append
 ↓
...

This is autoregressive generation.

And suddenly, streaming text from an LLM makes more sense too.

The application can expose generated tokens or chunks as they become available instead of waiting for the entire response to be generated.

My mental model now

Before:

Prompt → LLM → Answer

After this lecture:

Text
 ↓
Tokens
 ↓
Token IDs
 ↓
Embeddings + Position
 ↓
Transformer + Attention
 ↓
Contextual Representations
 ↓
Logits
 ↓
Probabilities
 ↓
Decoding
 ↓
Next Token
 ↓
Append
 ↓
Repeat

That is a much more useful mental model for me as an engineer.

It also gives me a better foundation for understanding the AI systems I'll eventually build around these models.

What I'm taking forward

The biggest lesson from this lecture isn't that I memorized definitions for:

Tokens
Embeddings
Attention
Transformers
Logits
Softmax
Temperature

It's that these concepts fit together into one generation process.

An LLM repeatedly uses context to predict what token should come next.

Everything else helps make that process possible and useful.

I'm continuing to study Forward Deployed Engineering and Generative AI alongside System Design.

The next step is to keep moving from:

understanding the model → understanding AI applications → understanding production AI systems.

I Started Learning Forward Deployed Engineering — And My First Lesson Was to Question the Requirement

Vikash Kumar — Tue, 20 Oct 2026 08:30:44 +0000

I recently started learning Forward Deployed Engineering alongside my System Design studies.

I expected the first lessons to be mostly about Generative AI, LLMs, tools and application architecture.

Instead, one of the first lessons was much more fundamental:

Before building the requested solution, understand the problem behind the request.

That sounds obvious.

But it changes how I think about engineering.
"We need an AI chatbot."
Imagine an e-commerce company tells an engineer:
"We need an AI chatbot."

The obvious reaction is to start thinking about:

Which LLM should we use?
Which framework?
How should we design the API?
Should we use RAG?
How should we write the prompts? But the request itself may not describe the actual problem.

The company might actually be experiencing:

Long customer-support resolution times
Repetitive work for support executives
Inconsistent answers
Increasing support costs
Too many routine requests
Lack of support outside business hours So the real requirement could be something closer to: Reduce the time required to resolve customer problems without reducing accuracy or customer trust.

Now the engineering problem looks different.

The chatbot is no longer the requirement.

It is one possible implementation.

Request vs requirement
This distinction was one of my biggest takeaways from the first FDE lesson.

A useful engineering flow is:
Customer Request
↓
Understand Business Problem
↓
Understand Existing Workflow
↓
Identify Constraints
↓
Design Technical Solution
↓
Integrate With Existing Systems
↓
Test & Evaluate
↓
Production
This is different from simply receiving a feature request and immediately implementing it.

The engineer has to discover what the system actually needs to accomplish.
Then comes another problem: the LLM cannot do everything
Suppose a customer says:
"Cancel my order and refund my payment."

An LLM can generate a convincing sentence:
"Your order has been cancelled."

But the sentence itself doesn't cancel an order.

A production system needs something more like:
Customer Request
↓
LLM understands intent
↓
Application validates request
↓
Cancellation API
↓
Order / Database System
↓
Confirmed Result
↓
Customer Response
Now we have engineering questions.

Is the customer authorized?

Has the order already shipped?

Is the order actually cancellable?

What happens if the API fails?

What if the model identifies the wrong order?

Should a large refund require human approval?

How do we audit the operation?

How do we test it?

How much latency is acceptable?

How much does each interaction cost?

The LLM is only one component of the system.
An LLM doesn't automatically know the company
A general-purpose LLM may understand concepts like refunds, payments and deliveries.

But it doesn't automatically know the company's current:

Refund policy
Customer order history
Delivery status
Payment records
Inventory
Internal procedures
Customer permissions The application therefore has to provide relevant information and connect the model with the systems that contain it.

Conceptually:
Customer Question
+
Company Policy
+
Customer Data
+
Previous Conversation
↓
LLM
↓
Grounded Response / Action
This is where AI application engineering becomes much more interesting to me.

The challenge isn't simply "use an LLM."

The challenge is building everything around the LLM so that the overall system behaves reliably.

What FDE adds to this
From my first lesson, I understand Forward Deployed Engineering as being close to the environment where the real problem exists.

An FDE works with customers, business teams, operations and engineering teams to understand the workflow, identify the actual problem, build a solution and help take it into production.

That means the role sits at an interesting boundary:
Real Customer Problem
↕
Forward Deployed Engineer
↕
Product + Data + APIs + Engineering
The technical solution is only part of the work.

Understanding the problem is also engineering work.
A lesson I want to carry forward
One sentence from this learning session stayed with me:

Don't start with the technology. Start with the problem.

It is easy to become excited about LLMs, agents, RAG, tools and frameworks.

But a production system still needs to solve a real problem.

And that means understanding:

Problem → Workflow → Constraints → System → Integration → Evaluation → Production

I'm going to continue this FDE + Generative AI journey alongside my System Design learning.

The goal isn't simply to learn how to call an LLM API.

The goal is to learn how to turn a real-world problem into a reliable engineering system.

I'm Exploring Low Level Design , and SOLID Is Changing How I Look At Code

Vikash Kumar — Tue, 15 Sep 2026 14:17:27 +0000

I've started going deeper into Low-Level Design as part of my journey toward becoming a stronger software engineer.

My initial understanding of LLD was mostly around classes, objects, relationships, and design patterns.

But one thing became clear very quickly:

Good LLD isn't about writing more classes or using more design patterns.

It's about making code easier to change without creating unnecessary complexity.

That's where SOLID started making more sense to me.

SOLID isn't five rules to memorize

The five principles are:

Single Responsibility Principle
Open/Closed Principle
Liskov Substitution Principle
Interface Segregation Principle
Dependency Inversion Principle

But instead of memorizing their definitions, I'm trying to understand the problems they're trying to prevent.

1. SRP — How many reasons can make this class change?_

Imagine an Invoice class responsible for:

calculating the invoice
printing the invoice
saving it to the database

It may work perfectly today.

But these responsibilities can change independently.

A change in calculation logic shouldn't necessarily require touching printing logic.

A database change shouldn't necessarily affect invoice calculation.

That made the idea of SRP much clearer to me:

The important question isn't simply "How many things does this class do?"

It's "How many independent reasons can make this class change?"

2. OCP — What happens when a new requirement arrives?

Suppose an invoice can currently be saved to a database.

Tomorrow the requirement becomes:

"We also need to save invoices to files."

One approach is to keep adding methods to the existing class.

But as requirements continue to grow, the class becomes a collection of unrelated variations.

Using an abstraction allows different implementations to provide different behaviors.

This made OCP feel less like:

"Never modify existing code."

and more like:

"Design stable parts so that new behavior can be added without constantly changing them."

3. LSP — Inheritance is more than code reuse

The Liskov Substitution Principle was particularly interesting to me.

Imagine a Vehicle abstraction that assumes every vehicle has an engine.

A car fits.

A motorcycle fits.

But a bicycle doesn't.

If client code expects every Vehicle to provide engine-related behavior, substituting a bicycle can break the program's assumptions.

The problem isn't simply that the child class has a different implementation.

The problem is that the parent abstraction made a promise that the child cannot satisfy.

That gave me a better way to think about inheritance:

A subtype should preserve the behavioral expectations of its parent.

4. ISP — Don't make clients implement what they don't need

Imagine one RestaurantEmployee interface containing:

washDishes()
serveCustomers()
cookFood()

A waiter shouldn't need to implement cooking and dishwashing just because those methods happen to exist on a large interface.

Breaking the interface into smaller, focused interfaces makes the dependency more meaningful.

The lesson:

An abstraction should be focused enough that its clients actually need what it exposes.

5. DIP — Depend on capability, not implementation

What I'm taking away

The biggest change in my thinking isn't that I now know what SOLID stands for.

It's that I'm starting to look at code by asking:

How easily can this code change?

When I see a class, I want to ask:

What can cause it to change?
Is it taking too many responsibilities?
What happens when a new behavior is introduced?
Can its subclasses really substitute for it?
Are clients depending on things they don't need?
Am I coupling the design to implementation details?

I still have a lot to learn.

Next, I'm going deeper into design patterns and LLD problems to understand how these principles translate into actual designs and code.

I'm documenting that journey as I go.

Consider a MacBook that directly creates:

WiredKeyboard
WiredMouse

The class now knows too much about concrete implementations.

If we instead make it depend on:

Keyboard
Mouse

we can provide different implementations without changing the MacBook itself.

This is where dependency inversion starts becoming practical:

High-level logic shouldn't be tightly coupled to low-level implementation details.