Business problem
- An AWS migration at risk of slipping
- No in-house skills for mature SQL Server to PostgreSQL conversion
- A need to prove nothing changed in translation
CASE STUDIES
Each one became a reusable pattern. Where a result is qualitative we say so, rather than inventing a percentage.
100%
objects migrated: Coal LSL
99.9%
data integrity: MetLife
By default
retention compliance: The Smith Family
1 search
across two systems: The Smith Family
100%
Database objects migrated, validated and deployed. Delivered in exactly three months; the AWS migration kicked off on time.
SQL Server · AWS · PostgreSQL · SQL · Noviq Shift
Cloud Strategy · Data Advisory
“Impressed with the extensive quality of the documentation and reconciliation reports… helped Coal LSL kick off their AWS migration on time.”
EXHIBIT 01
SCOPE MAP
Your cloud or platform partner moves the infrastructure and runs it well. The three layers above it, conversion and engineering, data trust, and the business outcome, are the ones your auditors, analysts and AI teams actually depend on, and they are typically nobody’s job.
Trusted reporting, analytics, knowledge search, AI use cases
PLATFORM PRACTICE
Not in scope
WITH THE DATA GUYZ
TDG lead
Reconciliation, data quality, PII, retention, lineage, evidence packs
PLATFORM PRACTICE
Not in scope
WITH THE DATA GUYZ
TDG lead
Hive / SQL / ETL conversion, Medallion lakehouse, pipelines, metadata
PLATFORM PRACTICE
Partial, infra-led
WITH THE DATA GUYZ
TDG lead, practice supports
Spark / Flink, Snowflake, Databricks, BigQuery, Dremio, MongoDB
PLATFORM PRACTICE
Practice configures
WITH THE DATA GUYZ
Shared: TDG designs the data model
Landing zone, Kerberos / Ranger → IAM / Unity Catalog, networking
PLATFORM PRACTICE
Practice lead
WITH THE DATA GUYZ
TDG supports
Cluster health, tuning, patching, monitoring, incident response
PLATFORM PRACTICE
Practice lead
WITH THE DATA GUYZ
None
So what: The migration can succeed on the platform practice’s own measures and still leave the client without a number they trust. That gap is the engagement.
Analytical mapping of a reference platform practice’s published scope against client needs; indicative.
EXHIBIT 02
TRANSITION MATRIX
Risk is highest wherever business logic is embedded in code rather than configuration, the top three rows. Those are the moves where conversion assurance and Noviq Shift earn their fee; the lower rows are largely tooling-led.
FROM
Hadoop (CDH / HDP / MapR)
HIGH CONVERSION RISK
TYPICAL TARGET
Databricks, Spark on EMR / Dataproc, Snowflake
PRACTICE OWNS
Cluster migration, security, landing zone
WE OWN
Hive and job conversion, reconciliation, Medallion, DQ
FROM
On-prem warehouse (Teradata, Oracle, SQL Server)
HIGH CONVERSION RISK
TYPICAL TARGET
Snowflake, BigQuery, Databricks SQL, Fabric
PRACTICE OWNS
Platform provisioning, network, identity
WE OWN
SQL and stored-procedure conversion, reconciliation, semantic layer
FROM
Legacy ETL (Informatica, SSIS, DataStage)
HIGH CONVERSION RISK
TYPICAL TARGET
ADF, Databricks Workflows, dbt, Matillion, Openflow
PRACTICE OWNS
Orchestration platform setup
WE OWN
Pipeline re-engineering, metadata-driven frameworks, load audit
FROM
Batch-only pipelines
TYPICAL TARGET
Flink, Spark Structured Streaming, Kafka
PRACTICE OWNS
Streaming infrastructure, brokers
WE OWN
Stream schema governance, quality checks, event data products
FROM
Query / BI layer (Impala, Presto, cubes)
TYPICAL TARGET
Dremio, Databricks SQL, BigQuery BI Engine
PRACTICE OWNS
Engine deployment and tuning
WE OWN
Semantic layer, Power BI / Looker models, analyst enablement
FROM
NoSQL / operational stores (HBase, Cassandra)
TYPICAL TARGET
MongoDB Atlas, DynamoDB, Cosmos DB
PRACTICE OWNS
Database migration and scaling
WE OWN
Schema governance, PII controls, analytics offload
FROM
Siloed knowledge (files, wikis, tickets)
TYPICAL TARGET
Snowflake Cortex, BigQuery with Gemini, vector search
PRACTICE OWNS
Platform access and capacity
WE OWN
Ingestion pipelines, governed metadata, AI-ready search
So what: Scope the top three rows as fixed-price conversion assurance with reconciliation evidence. Scope the rest as engineering. Pricing them the same way is how migration programmes overrun.
Transition patterns and risk ratings are our analytical view for scoping; targets vary by client platform choice.
99.9%
Data integrity and auditability. MetLife owns and extends the rules; every layer reconciled. Manual effort significantly reduced.
Snowflake · Matillion · SQL · Medallion architecture
Data Advisory · Insights & Analytics
“Their deep technical expertise and proactive problem-solving made a measurable impact on our data ecosystem. We'd be happy to recommend The Data Guyz.”
Compliant by default, with audit-ready reporting.
Azure Synapse · Azure SQL · Microsoft Fabric · Power BI
Data Advisory · Insights & Analytics
AI-ready knowledge in one search, with nothing bespoke to maintain.
Snowflake Cortex Search · Openflow · ADLS · Zendesk · SharePoint
AI Adoption · Insights & Analytics
EXHIBIT 03
WHY RETRIEVAL FAILS
The most common AI complaint from enterprises that have already invested in data. There are four root causes. Two are configuration and two are data work, and the fix for one does nothing for the others, which is why we test before we recommend.
CAUSE 01 · CONFIGURATION
Missing indexes, restricted permissions, repositories never connected.
Test: can the affected user discover a file we know exists?
CAUSE 02 · DATA WORK
Scanned PDFs, unreadable text, inconsistent formats, missing metadata. An agent cannot read what was never extracted.
Test: extraction accuracy on the clauses that matter.
CAUSE 03 · CONFIGURATION
Irrelevant results, missed clauses, too little context handed to the model to answer with.
Test: correct document and clause retrieval rate.
CAUSE 04 · DATA WORK
Duplicate contracts, missing amendments, unclear status. The agent answers from v1 while the business runs on vFINAL_edited.
Test: does it retrieve the latest effective agreement?
So what: replacing the assistant fixes causes 01 and 03 at best. If the diagnosis lands on 02 or 04, a new product changes nothing, and that is roughly half the cases we see. Diagnose against 20–30 benchmark questions before anyone signs a replacement.
Diagnostic framework from our enterprise document-retrieval practice. Scoped as the Knowledge-Engine Diagnostic.
EXHIBIT 04
THE KNOWLEDGE ENGINE
What we build when the diagnosis says build. Every layer sits inside the enterprise security and role-based access boundary, so retrieval respects permissions rather than having them bolted on afterwards.
LAYER 01 · SOURCES
Document sources
SharePoint, PDFs, Word, ticketing systems, wikis, shared drives, connected, not copied.
LAYER 02 · THE STEP MOST TEAMS SKIP
Processing engine
OCR · extraction · clause-aware chunking · metadata generation. This layer decides everything downstream of it.
LAYER 03 · INSIDE THE SECURITY & RBAC BOUNDARY
Hybrid search
Keyword match plus vector search plus reranking, because neither alone finds a clause worded differently to the question.
LAYER 04 · INSIDE THE SECURITY & RBAC BOUNDARY
Generation with citations
Synthesised answers with deterministic source citations, file and clause, every time, or no answer at all.
LAYER 05 · WHERE PEOPLE ASK
Search portal or agent
A portal, Teams, or an agent with a bounded tool list and an approval gate.
WHAT CLAUSE-LEVEL RETRIEVAL LOOKS LIKE
“What is the termination notice period for ABC Pty Ltd?” → 90 days’ written notice. Source: Service_Agreement_ABC_2025.pdf, Clause 12.3.
Illustrative example. Not actual client data or validated results.
So what: the expensive layer is 02, not the model. Teams that buy a better model and skip extraction and chunking spend the budget and keep the problem.
EXHIBIT 05
AGENTS ON EXISTING BI
The second agent pattern. A lakehouse and several hundred reports already exist, and an executive who wants one number still waits on an analyst. The fix is an agent on the BI you already own, not a platform rebuild, and not at the cost of the reports people rely on.
01 / KEEP WHAT WORKS
For repeatable decisions
The right recurring reports, clear ownership, consistent business measures. We rationalise; we don’t delete.
02 / MAKE IT EASIER
For flexible exploration
Business teams explore approved information without starting from scratch or raising a ticket.
03 / BUILD THE NEW LAYER
For questions and alerts
Ask, explain, find and monitor, in everyday language, answered from approved metrics with the source report named.
Controlled end to end: security · approved definitions · data quality · monitoring · business ownership
01 DESIGN
The valuable business questions, the user journeys and the right agent skills. A short honest list beats an agent claiming to do everything.
02 CONNECT
Existing reports, approved measures and catalogue context, so the agent is right rather than plausible.
03 MAKE IT SAFE
Permissions enforced at retrieval, answers tested against a golden set, unsupported questions handled clearly instead of guessed at.
04 EMBED
Launch with real users, monitor usefulness, refine against evidence, then roll the proven skills wider.
So what: measure the pilot on answer quality, time to insight, adoption, reporting reuse and operational cost. An agent nobody opens twice is a failed project however good the demo was.
Pattern developed for a quick-service restaurant group’s reporting environment. Scoped as the Insight Agent Pilot.
WHAT COMPOUNDS
A consultancy that leaves nothing behind has to rebuild its margin on every deal, and you pay for the rebuild. Each engagement below produced a reusable pattern that now ships as part of the next one.
So what: ask what the last three engagements left behind. If the answer is a report, you are funding a first build every time. If it is a pattern, you are buying the third version of something.
INSIGHTS & GUIDES
Twelve guides on data migration, reconciliation, data quality, privacy and AI data readiness. Each one is built on work we have actually delivered, and names the client engagement that proves it. Expand to read.
A superannuation-sector client had an AWS migration at risk of slipping. The platform partner could move the infrastructure; nobody on the team had converted a mature SQL Server estate to PostgreSQL before, and nobody could prove the business logic would survive the translation.
The article walks through the method: building the object and dependency inventory first, converting tables, views, stored procedures and functions in dependency order, and running source-to-target reconciliation after every wave rather than once at the end. It covers what converts mechanically, what needs a human (cursors, dynamic SQL, proprietary date handling), and what the reconciliation evidence pack has to contain before an auditor will accept a cutover.
Outcome: 100% of database objects converted, validated and deployed in three months, with audit-ready reconciliation at every step. The AWS migration kicked off on time.
Moving the data is the easy half. The hard half is proving to a business owner, an auditor or a regulator that what arrived is what left.
A working checklist: row-count reconciliation per layer, key-level comparison on the columns that carry business meaning, transformation-rule validation against the specification rather than against the code, exception analysis with a documented disposition for every break, and the evidence capture that turns all of it into something reviewable. Includes the reconciliation report template we hand over at cutover.
Holding data past its retention period is a compliance exposure that grows quietly. Manual disposal across platforms is slow, error-prone and almost impossible to evidence after the fact.
This covers the Automated Data Disposal and De-identification Framework: how retention rules are expressed as policy rather than scripts, how cascading disposal handles records referenced across systems, how de-identification keeps analytical value in data you are no longer allowed to hold in identified form, and the Power BI reporting that shows leadership what was disposed, what was de-identified and what is due next period.
Most data quality tools are a licence, a UI and a dependency on whoever configured it. Written in SQL instead, the rules live in the warehouse your team already operates, and the engineer who inherits them can read them.
Covers the rule taxonomy (completeness, validity, consistency, timeliness, uniqueness), how to make rules configuration-driven so a new table does not mean new code, where to put the results so they are visible to the business rather than buried in a log, and how the daily failure and remediation report gets read by someone who can act on it.
Publishing roughly two a month. Want one of the drafts early? Ask us.
THE EVIDENCE TRAIL
The deliverable in a data engagement is not the pipeline. It is the ability to answer a reviewer who asks where a number came from. These four artefacts are what makes that answerable, and they come out of every engagement on this page.
So what: ask any firm to show you a redacted version of all four from a previous engagement. If they cannot produce them, those artefacts were never built.
We can walk through the architecture, the reconciliation evidence and the handover pack from any of these four engagements, redacted where the client requires it.