ProCon Leicestershire · 16 September 2026

Beyond
the hype

One real AI project, told properly. What we built, what went wrong, what fixed it, and what we would want you to know before you start your own.

The case study

AI for tender intelligence

A client drowning in tender documents wanted three things out of them, quickly enough to decide whether a bid was worth chasing at all.

01

Structured summarisation

Pull out the specific things a bid team needs, milestone dates and commercial detail, rather than producing another generic summary nobody reads.

02

Bid scoring

Score each opportunity against the client's own criteria so the pipeline could be prioritised, and a bid or no-bid call made early.

03

Question extraction

Separate the front-end administrative questions from the deeper technical ones that need a subject expert to sit down and answer.

Warts and all

What went wrong

None of this appears in a vendor case study. All of it is normal, and all of it costs time and money if you meet it for the first time halfway through a build.

01

Large and complex documents

  • Documents varied wildly in format and structure
  • Token windows were too small to take a whole document
  • We had to chunk the text and aggregate what came back
  • No consistent template across tenders
  • A mix of PDFs, Word files and spreadsheets
  • Numbered lists, bullets and tables were the worst of it
02

Inconsistent performance

  • The stochastic nature of the model gave inconsistent extractions
  • The same prompt produced different results run to run
  • Questions found in one run were missed in the next
  • Confidence scores moved unpredictably
  • No guarantee of completeness or accuracy
  • Which made testing with the client genuinely hard
03

Refinement and overfitting

  • Fixing one prompt problem often broke another
  • Real risk of overfitting to the tender formats in front of us
  • It had to generalise to documents we had never seen
  • Regression testing was essential, and slow
  • Every improvement needed validating at scale
  • Prompt engineering is an art rather than a science
If you remember one thing

AI is powerful. It is not a silver bullet.

Every one of those problems was solved with ordinary engineering discipline rather than a better model. Careful implementation, validation and human oversight are what turned a promising demo into something a bid team could rely on.

And how we overcame them

What fixed it

Three techniques did most of the work. None of them is exotic, and all of them are worth budgeting for at the start rather than discovering halfway through.

01

Chunking and aggregation

Process the document in segments and merge the extracted results, with a moving window across the text so context is not lost at the seams.

02

Repeated runs and collation

Run the extraction several times and combine the outputs, excluding the outliers automatically. Three passes in the eighties and nineties collated to a materially better answer than any single run.

03

AI unit testing

The client marked up a set of perfect results. Every change was then checked against that baseline, using a mix of AI and plain regular expressions to validate. Without it you cannot tell an improvement from a regression.

The broader picture

Other risks to plan for

Not all of these bit us on this project. All of them are worth a conversation before you put an AI system anywhere near a client, a regulator or a tender response.

Data security

Where your documents go, and who trains on them.

Copyright and IP

What the model was trained on, and what you now own.

Bias and harmful content

Inherited from the training data rather than from you.

Hallucinations

Confident, fluent, and wrong.

Stochastic results

The same input need not give the same output.

Prompt injection

A document that instructs the model reading it.

Model cut-off dates

It does not know what happened after training.

Weak logical reasoning

Improving, still not arithmetic or formal logic.

Reputational damage

All of the above, in front of a client.

Conclusions

What we would tell you

1

AI is powerful, but not a silver bullet

Every case study we have run hit non-trivial challenges. Careful implementation, validation and oversight are the job, and not an afterthought.

2

Models improve, and the challenges remain

Better accuracy and lower costs have taken the edge off the early problems. Progress may be plateauing, which means smarter engineering rather than simply waiting for a better model.

3

You plus AI beats you

The best results come from AI augmenting people rather than replacing them. Explainability, validation and control are what make that work.

4

Autonomy raises the stakes

As the industry moves towards agentic AI, hallucinations, errors and unpredictability matter more rather than less. Governance, safeguards and human oversight become more important as systems become more autonomous.

5

A clear business case is essential

Adoption should be driven by a real business need rather than by the hype. Cost, feasibility and return belong in the conversation before implementation.

6

And the exciting part

We can now solve problems that were not solvable a few years ago, processing complex data at a scale and cost that simply was not available before.

The offer from the stage

Want a hand with your AI pilot?

Thirty minutes, no charge. Tell us where your pilot is stuck and we will tell you what we think, based on having been stuck there ourselves.

We build AI and data systems into construction, infrastructure and property businesses, and our engineers work inside your systems for as long as that is useful and no longer. We will tell you when you do not need us.

Tell us about your pilot

Alex reads these himself.