All services

Data quality for AI: one set of numbers from ERP, spreadsheets and mailboxes

In most businesses the data sits spread across ERP, spreadsheets and mailboxes, some of it twice over, some of it missing. We bring it into a shape AI can work with at all.

Numbers that exist, just not in one place

Most businesses have their numbers. Contribution margin sits in the ERP, case duration in the ticketing system, effort in time tracking, customer details in the mailbox, and once a quarter somebody joins them up in a spreadsheet. Some figures exist twice and disagree. Some do not exist at all.

So the important questions get estimated. Which order type does not pay for itself. Where a case loses its time. How often work has to be redone. You notice it when two department heads bring two different figures for the same thing to the same meeting, and the meeting is spent on which one is right.

The AI angle is the same problem seen from the other side. An AI project on data that is not maintained fails on the data, not on the model, and it fails late, after the build. Knowing the state of your data before that is cheaper than finding out after.

Assessing the data situation

Before anything is built, we write down the data situation for the question at hand, in plain language rather than in a schema. For each figure: where it lives, who maintains it, how often, whether it exists somewhere else as well and whether the two agree, and what is missing altogether.

The result is one of three verdicts per figure: usable now, usable after a named clean-up, or not there. That page is what a decision maker needs to decide whether the question is worth pursuing, and it is the same page the assessment produces for every AI candidate.

How we work

  1. Fix the question, sharply enough that a number can answer it. “How is service doing” is not a question. “Which service contracts cost more hours than they bill” is.
  2. Find the sources the question needs and connect them, usually through an export or a read-only interface. Reconcile where they disagree, and write down the disagreement rather than averaging it away.
  3. Define the metrics once, in writing, the same for everyone. Which hours count, which orders are excluded, which date is the cut-off.
  4. Build the view somebody can read without an explanation, in the tool you already use where possible.
  5. State the caveats on the figure itself. A number without its uncertainty is a claim.
  6. Only then a shared foundation, when the same sources start answering several questions and the reconciliation would otherwise be done twice.

A worked example: which order type pays

A contract manufacturer in North Rhine-Westphalia wants to know which order types do not cover their costs. The revenue per order is in the ERP. The hours are in time tracking, per project. The rework is in the ticketing system. The three systems share a project number, except that it is written three different ways, and in one of them it is often left blank.

The work is the mapping: a rule that reconciles the three spellings, a rule for the blanks, and a note on the share of hours that could not be assigned to any order. The view shows margin per order type with that caveat printed next to it. The argument in the management meeting ends, because the figure and its uncertainty are on the same page.

The follow-up costs nothing to build: one project number, written one way, entered at the source. The next analysis will have a smaller caveat.

What you need to bring, and when it is not worth it

You need read access to the systems, or exports. You need somebody who knows what the fields actually mean, because the column called “status” rarely means what it says. And you need willingness to change habits at the point of entry, because that is where data quality is made and no analysis can repair it afterwards.

It is not worth it for a one-off question that a query against one system answers in an hour; we will tell you which query. It is not worth it when the data simply does not exist yet; then the first step is to collect it, and that is a process change, not an analytics project. And it is not worth it when the reporting module in your ERP already answers the question and nobody has opened it.

How it differs from the neighbouring services

Data and analytics answers questions with figures from your systems. A knowledge system answers with passages from your documents. The assessment records the data situation for each AI candidate; this service is what fixes it. Process automation is often where the better data comes from, because a rule enters the project number the same way every time and a person does not.

The next step

In the free introductory call, bring the figure people argue about most. It is usually the right place to start.

Common questions

Do we need a data warehouse for this?
Not at the start. The first analysis usually comes from two or three sources and answers one question. A warehouse earns its place when several departments calculate the same figures differently. Then it is the solution, not the entry point.
Our data is messy. Does that disqualify us?
No, but it is work with a cost. We state what is missing, what exists twice and what contradicts itself before building. An analysis on unclear data is worse than none, because people believe it.
What does "ready for AI" actually mean?
Four things, and none of them is a platform. Fields that are filled in the same way every time. One place where each fact lives, rather than three that disagree. History that is kept rather than overwritten. And documents stored where somebody owns them. Most of that is habit at the point of entry, not technology.
What does this have to do with AI?
Two things. First, AI projects regularly fail on the data situation rather than the model. Second, most questions AI gets proposed for are answered better by a query: faster, cheaper and traceable. We say which is which.
Where does the data live?
In your systems, or in an environment that belongs to you. Where we process it, that happens in German data centres under a data processing agreement under Article 28 GDPR, or inside your own infrastructure.