What Have We Learned About Semantic Models and AI Accuracy?

At the end of the day, what we learned is a semantic model is to get any accurate

answer. If you don't have good semantic models, you will not get any good answers.

LLMs are impressive for sure, but they generate language.

They don't do mathematical reasoning, they don't have logical reasoning, and

that's exactly why you have to pair them with a deterministic engine.

If your context is subpar, the quality of the answer will be subpar.

You will not have any chance of getting high-quality answers from the LLM if you do

not have a good context. The reason is even though the calculation engine is

involved, at the end of the day, inputs to the calculation, the code of

the right dataset API, the filters that it passes to the dataset API, it's

something that the LLM will need come up with.

And if the context is not correct, the LLM might interpret a certain column or a

certain dimension in a cube wrong. And so the naming is very important.

Explaining to the LLM what each name means, which is the meaning of that dimension.

If you do, for example, any FDH calculation, if you call margin a certain

dimension and then you call it net revenue in another place, I mean, the LLM will

know how to add them correctly. You really need to make sure that you are

consistent in naming convention because the application, in many cases, has grown

organically. So you implement one functionality, then a different one, then another

one, and you stack on multiple data cubes, multiple tables, multiple data sources.

If things are inconsistent, you need to do a bit of organizational work to give the

LLM dictionary so the LLM does not hallucinate.

Because again, even though the calculations are always deterministic, it doesn't

mean that the LLM is not able to give the wrong input to the calculation.

And that's where you really need to

focus .