Documents: find the passages
Policies, contracts, manuals, tickets. The AI retrieves the relevant passages and writes an answer from them, citing the source. The technical term is retrieval-augmented generation (RAG).
Insights · AI architecture
Chat with your own data: what it takes
Ask a question in plain language and get an answer from your own systems, with sources: a language model cannot do this on its own. It takes clear questions, data that can be found, agreed definitions and permissions that carry through to the answer.
By Michele Mangieri · Updated
In short
Chatting with your own data means asking a question in plain language and getting an answer from your own systems, with sources. A language model cannot do this on its own. It takes clear questions, data that can be found, agreed definitions and permissions that carry through to the answer. Documents and figures need different paths.
Two kinds of data
Documents and figures need different paths.
Figures: generate queries
Revenue, stock levels, delivery dates from ERP, CRM or a data platform. The AI translates the question into a query on approved metrics and does not calculate on its own. Every result stays verifiable.
Both together
Many real questions mix the two, for example: “Which suppliers with a framework agreement are behind on delivery dates?” An orchestration layer splits the question into parts and combines the answers.
Eight steps
From the question to an answer with sources.
- Clarify the questions: who asks what, how often, and what does a good answer look like? Twenty real questions from the business are the best starting point.
- Find the data: where do the answers live today? Which system is the source of truth, and how current and complete is the data?
- Agree on definitions: what counts as “revenue”, an “active customer”, an “open order”? Without a shared glossary and defined metrics, the AI gives different figures for the same question.
- Pass permissions through: the AI sees only what the person asking may see. Permissions of the source systems apply all the way to the answer, instead of one shared account for everyone.
- Connect: documents are prepared and made searchable; structured data is connected through interfaces or approved views. Copies only where there is no other way.
- Set guardrails: every answer names its source. If data is missing, the AI says so instead of guessing. Confidential and personal data stays protected.
- Test with real questions: the twenty questions from step one become the test bench. Business experts rate the answers before a wide rollout.
- Operate: track usage, cost, answer quality and feedback continuously. New sources are added following the same pattern.
What Gartner expects
Data decides, not the model.
60%of AI projects without AI-ready data will be abandoned through 2026, according to Gartner
63%of organizations surveyed either lack the right data management practices for AI or are unsure
AI-ready data does not mean “as much data as possible”. It means data whose origin, meaning and quality are proven for the specific use case.
Source: Gartner, press release “Lack of AI-Ready Data Puts AI Projects at Risk”, 26 February 2025, including a survey of around 1,200 data management leaders. Gartner is a registered trademark of Gartner, Inc. and/or its affiliates.
What matters
A chat is quick to build. Trust takes longer.
- Start small: one business area, one group of questions, a few sources. Breadth comes after the proof.
- Definitions before technology: most wrong answers come from unclear definitions, not from the model.
- Name owners: for every source it is clear who is responsible for content and currency.
- Stay replaceable: language models change fast. If data, definitions and permissions are kept separate from the model, you can swap the model without rebuilding.
Read on
How the data chat fits into an overall architecture is covered in AI reference architecture. An example from one business area: AI in procurement. How AI is connected to your systems is explained in MCP server explained simply.
Want to know whether your data is ready for an AI chat? AI Decision Readiness →
Frequently asked questions
Questions about chatting with your own data
How does an AI chat answer questions about documents?
Through retrieval-augmented generation (RAG). The AI retrieves the relevant passages from policies, contracts, manuals or tickets and writes an answer from them, citing the source.
Does the AI calculate figures from ERP or CRM itself?
No. The AI translates the question into a query on approved metrics and does not calculate on its own, so every result stays verifiable. When a question mixes documents and figures, an orchestration layer splits it into parts and combines the answers.
How do you start a chat with your own data?
Start small: one business area, one group of questions, a few sources. Twenty real questions from the business are the best starting point. They later become the test bench on which business experts rate the answers before a wide rollout.
What does AI-ready data mean?
Not as much data as possible, but data whose origin, meaning and quality are proven for the specific use case. Gartner expects 60% of AI projects without AI-ready data to be abandoned through 2026.
30 minutes, and you will know whether we can help.
You describe the decision ahead. We tell you which package fits, or that none does.