The Data Guyz logo The Data Guyz YOUR DATA EXPERTS Talk to us

CASE STUDIES

Four engagements. The problem, the build, the number.

Each one became a reusable pattern. Where a result is qualitative we say so, rather than inventing a percentage.

100%

objects migrated: Coal LSL

99.9%

data integrity: MetLife

By default

retention compliance: The Smith Family

1 search

across two systems: The Smith Family

Coal LSL CLOUD STRATEGY · MIGRATE

SQL Server to AWS PostgreSQL: 100% of objects in three months.

Business problem

  • An AWS migration at risk of slipping
  • No in-house skills for mature SQL Server to PostgreSQL conversion
  • A need to prove nothing changed in translation

Solution: what we built

  • Full object conversion: every table, view, stored procedure and function
  • Rigorous validation of each object and dataset against source
  • Detailed, audit-ready reconciliation reporting at every step

Outcome

100%

Database objects migrated, validated and deployed. Delivered in exactly three months; the AWS migration kicked off on time.

TECH STACK

SQL Server · AWS · PostgreSQL · SQL · Noviq Shift

PILLARS

Cloud Strategy · Data Advisory

“Impressed with the extensive quality of the documentation and reconciliation reports… helped Coal LSL kick off their AWS migration on time.”

Tim Gardner
Data Platform Manager, Coal LSL
METHOD BEHIND THIS ENGAGEMENT How we scope a platform migration, and where the conversion risk sits Two working views from this engagement: the six-layer scope map that separates our work from the platform partner’s, and the transition matrix that prices conversion risk by legacy estate.

EXHIBIT 01

SCOPE MAP

Three of six layers sit outside a platform practice’s scope.

Your cloud or platform partner moves the infrastructure and runs it well. The three layers above it, conversion and engineering, data trust, and the business outcome, are the ones your auditors, analysts and AI teams actually depend on, and they are typically nobody’s job.

Business outcome & AI

Trusted reporting, analytics, knowledge search, AI use cases

PLATFORM PRACTICE

Not in scope

WITH THE DATA GUYZ

TDG lead

Data trust & governance

Reconciliation, data quality, PII, retention, lineage, evidence packs

PLATFORM PRACTICE

Not in scope

WITH THE DATA GUYZ

TDG lead

Data migration & engineering

Hive / SQL / ETL conversion, Medallion lakehouse, pipelines, metadata

PLATFORM PRACTICE

Partial, infra-led

WITH THE DATA GUYZ

TDG lead, practice supports

Target platform

Spark / Flink, Snowflake, Databricks, BigQuery, Dremio, MongoDB

PLATFORM PRACTICE

Practice configures

WITH THE DATA GUYZ

Shared: TDG designs the data model

Cloud infrastructure & security

Landing zone, Kerberos / Ranger → IAM / Unity Catalog, networking

PLATFORM PRACTICE

Practice lead

WITH THE DATA GUYZ

TDG supports

Legacy platform & managed run

Cluster health, tuning, patching, monitoring, incident response

PLATFORM PRACTICE

Practice lead

WITH THE DATA GUYZ

None

So what: The migration can succeed on the platform practice’s own measures and still leave the client without a number they trust. That gap is the engagement.

Analytical mapping of a reference platform practice’s published scope against client needs; indicative.

EXHIBIT 02

TRANSITION MATRIX

Where the conversion risk actually sits.

Risk is highest wherever business logic is embedded in code rather than configuration, the top three rows. Those are the moves where conversion assurance and Noviq Shift earn their fee; the lower rows are largely tooling-led.

FROM

Hadoop (CDH / HDP / MapR)

HIGH CONVERSION RISK

TYPICAL TARGET

Databricks, Spark on EMR / Dataproc, Snowflake

PRACTICE OWNS

Cluster migration, security, landing zone

WE OWN

Hive and job conversion, reconciliation, Medallion, DQ

FROM

On-prem warehouse (Teradata, Oracle, SQL Server)

HIGH CONVERSION RISK

TYPICAL TARGET

Snowflake, BigQuery, Databricks SQL, Fabric

PRACTICE OWNS

Platform provisioning, network, identity

WE OWN

SQL and stored-procedure conversion, reconciliation, semantic layer

FROM

Legacy ETL (Informatica, SSIS, DataStage)

HIGH CONVERSION RISK

TYPICAL TARGET

ADF, Databricks Workflows, dbt, Matillion, Openflow

PRACTICE OWNS

Orchestration platform setup

WE OWN

Pipeline re-engineering, metadata-driven frameworks, load audit

FROM

Batch-only pipelines

TYPICAL TARGET

Flink, Spark Structured Streaming, Kafka

PRACTICE OWNS

Streaming infrastructure, brokers

WE OWN

Stream schema governance, quality checks, event data products

FROM

Query / BI layer (Impala, Presto, cubes)

TYPICAL TARGET

Dremio, Databricks SQL, BigQuery BI Engine

PRACTICE OWNS

Engine deployment and tuning

WE OWN

Semantic layer, Power BI / Looker models, analyst enablement

FROM

NoSQL / operational stores (HBase, Cassandra)

TYPICAL TARGET

MongoDB Atlas, DynamoDB, Cosmos DB

PRACTICE OWNS

Database migration and scaling

WE OWN

Schema governance, PII controls, analytics offload

FROM

Siloed knowledge (files, wikis, tickets)

TYPICAL TARGET

Snowflake Cortex, BigQuery with Gemini, vector search

PRACTICE OWNS

Platform access and capacity

WE OWN

Ingestion pipelines, governed metadata, AI-ready search

So what: Scope the top three rows as fixed-price conversion assurance with reconciliation evidence. Scope the rest as engineering. Pricing them the same way is how migration programmes overrun.

Transition patterns and risk ratings are our analytical view for scoping; targets vary by client platform choice.

MetLife DATA ADVISORY · GOVERN

Automated data quality and load audit: 99.9% integrity.

Business problem

  • Growing data volumes across a global insurer
  • No way to answer: can we trust this data, and did every record make it?
  • No organisation-wide view of where sensitive PII sits

Solution: what we built

  • A scalable data-quality framework built entirely in SQL, so MetLife owns and extends the rules
  • A load audit framework on Matillion surfacing source-versus-target counts at every layer
  • An organisation-wide data dictionary and PII classification framework

Outcome

99.9%

Data integrity and auditability. MetLife owns and extends the rules; every layer reconciled. Manual effort significantly reduced.

TECH STACK

Snowflake · Matillion · SQL · Medallion architecture

PILLARS

Data Advisory · Insights & Analytics

“Their deep technical expertise and proactive problem-solving made a measurable impact on our data ecosystem. We'd be happy to recommend The Data Guyz.”

Srilata Nutakki
MetLife
The Smith Family COMPLY

ADDF: automated data disposal and de-identification

Business problem

  • Keeping data longer than necessary created regulatory exposure, and manual disposal across multiple platforms was slow, error-prone and hard to prove

Solution: what we built

  • Automated scheduled purging at the end of retention periods
  • Removal of personal identifiers from data kept for analysis
  • Power BI reporting on disposal and de-identification status

Outcome

Compliant by default, with audit-ready reporting.

TECH STACK

Azure Synapse · Azure SQL · Microsoft Fabric · Power BI

PILLARS

Data Advisory · Insights & Analytics

The Smith Family AI ADOPTION

Snowflake Cortex knowledge search

Business problem

  • Knowledge was spread across Zendesk knowledge-base articles and SharePoint files, so finding an answer meant searching several systems

Solution: what we built

  • Zendesk content loaded into ADLS and indexed in Cortex Search
  • SharePoint files brought in via Openflow, with no custom integration layer
  • The pipeline tuned end to end, from ingestion through to indexing

Outcome

AI-ready knowledge in one search, with nothing bespoke to maintain.

TECH STACK

Snowflake Cortex Search · Openflow · ADLS · Zendesk · SharePoint

PILLARS

AI Adoption · Insights & Analytics

METHOD BEHIND THIS ENGAGEMENT Why enterprise search fails, what we build instead, and where it goes next The four root causes we test for before recommending anything, the retrieval architecture we built here, and the insight-agent pattern this foundation makes possible.

EXHIBIT 03

WHY RETRIEVAL FAILS

“We bought Copilot and it still can’t find the contract.”

The most common AI complaint from enterprises that have already invested in data. There are four root causes. Two are configuration and two are data work, and the fix for one does nothing for the others, which is why we test before we recommend.

CAUSE 01 · CONFIGURATION

Incomplete indexing & access

Missing indexes, restricted permissions, repositories never connected.

Test: can the affected user discover a file we know exists?

CAUSE 02 · DATA WORK

Poor document quality & structure

Scanned PDFs, unreadable text, inconsistent formats, missing metadata. An agent cannot read what was never extracted.

Test: extraction accuracy on the clauses that matter.

CAUSE 03 · CONFIGURATION

Ineffective semantic retrieval

Irrelevant results, missed clauses, too little context handed to the model to answer with.

Test: correct document and clause retrieval rate.

CAUSE 04 · DATA WORK

Versioning & governance

Duplicate contracts, missing amendments, unclear status. The agent answers from v1 while the business runs on vFINAL_edited.

Test: does it retrieve the latest effective agreement?

So what: replacing the assistant fixes causes 01 and 03 at best. If the diagnosis lands on 02 or 04, a new product changes nothing, and that is roughly half the cases we see. Diagnose against 20–30 benchmark questions before anyone signs a replacement.

Diagnostic framework from our enterprise document-retrieval practice. Scoped as the Knowledge-Engine Diagnostic.

EXHIBIT 04

THE KNOWLEDGE ENGINE

The pipeline, the metadata and the retrieval: in that order.

What we build when the diagnosis says build. Every layer sits inside the enterprise security and role-based access boundary, so retrieval respects permissions rather than having them bolted on afterwards.

LAYER 01 · SOURCES

Document sources

SharePoint, PDFs, Word, ticketing systems, wikis, shared drives, connected, not copied.

LAYER 02 · THE STEP MOST TEAMS SKIP

Processing engine

OCR · extraction · clause-aware chunking · metadata generation. This layer decides everything downstream of it.

LAYER 03 · INSIDE THE SECURITY & RBAC BOUNDARY

Hybrid search

Keyword match plus vector search plus reranking, because neither alone finds a clause worded differently to the question.

LAYER 04 · INSIDE THE SECURITY & RBAC BOUNDARY

Generation with citations

Synthesised answers with deterministic source citations, file and clause, every time, or no answer at all.

LAYER 05 · WHERE PEOPLE ASK

Search portal or agent

A portal, Teams, or an agent with a bounded tool list and an approval gate.

WHAT CLAUSE-LEVEL RETRIEVAL LOOKS LIKE

“What is the termination notice period for ABC Pty Ltd?” → 90 days’ written notice. Source: Service_Agreement_ABC_2025.pdf, Clause 12.3.

Illustrative example. Not actual client data or validated results.

So what: the expensive layer is 02, not the model. Teams that buy a better model and skip extraction and chunking spend the budget and keep the problem.

EXHIBIT 05

AGENTS ON EXISTING BI

Three ways to get an answer, one source of truth underneath.

The second agent pattern. A lakehouse and several hundred reports already exist, and an executive who wants one number still waits on an analyst. The fix is an agent on the BI you already own, not a platform rebuild, and not at the cost of the reports people rely on.

01 / KEEP WHAT WORKS

Trusted reports

For repeatable decisions

The right recurring reports, clear ownership, consistent business measures. We rationalise; we don’t delete.

02 / MAKE IT EASIER

Governed self-service

For flexible exploration

Business teams explore approved information without starting from scratch or raising a ticket.

03 / BUILD THE NEW LAYER

The insight agent

For questions and alerts

Ask, explain, find and monitor, in everyday language, answered from approved metrics with the source report named.

Controlled end to end: security · approved definitions · data quality · monitoring · business ownership

01 DESIGN

The valuable business questions, the user journeys and the right agent skills. A short honest list beats an agent claiming to do everything.

02 CONNECT

Existing reports, approved measures and catalogue context, so the agent is right rather than plausible.

03 MAKE IT SAFE

Permissions enforced at retrieval, answers tested against a golden set, unsupported questions handled clearly instead of guessed at.

04 EMBED

Launch with real users, monitor usefulness, refine against evidence, then roll the proven skills wider.

So what: measure the pilot on answer quality, time to insight, adoption, reporting reuse and operational cost. An agent nobody opens twice is a failed project however good the demo was.

Pattern developed for a quick-service restaurant group’s reporting environment. Scoped as the Insight Agent Pilot.

WHAT COMPOUNDS

Four engagements. Four patterns the next client inherits.

A consultancy that leaves nothing behind has to rebuild its margin on every deal, and you pay for the rebuild. Each engagement below produced a reusable pattern that now ships as part of the next one.

FOUR ENGAGEMENTS: AND WHAT EACH ONE LEFT BEHIND MIGRATECoal LSL100%of objects converted,validated, deployedPATTERN INHERITEDConversion assuranceGOVERNMetLife99.9%data integrity acrossevery layerPATTERN INHERITEDPost-migration trust layerCOMPLYThe Smith FamilyBy defaultretention compliance,manual effort removedPATTERN INHERITEDGovernance on the lakehouseACTIVATEThe Smith Family1 searchover Zendesk andSharePoint knowledgePATTERN INHERITEDAI readiness after migration Each pattern was paid for once and is inherited by every client after it. That is why the second engagement of a kind is cheaper than the first. Where an outcome is qualitative we say so rather than inventing a percentage.
Top row is the delivered outcome; bottom row is the asset it became. The arrow is the part most firms do not do.

So what: ask what the last three engagements left behind. If the answer is a report, you are funding a first build every time. If it is a pattern, you are buying the third version of something.

INSIGHTS & GUIDES

Written from the engagements, not from a keyword tool.

Twelve guides on data migration, reconciliation, data quality, privacy and AI data readiness. Each one is built on work we have actually delivered, and names the client engagement that proves it. Expand to read.

DATABASE CONVERSION How we rescued a stalled SQL Server to PostgreSQL migration on AWS in 3 months Target keywords: SQL Server to PostgreSQL migration · stalled data migration recovery  ·  Proof: Coal LSL

A superannuation-sector client had an AWS migration at risk of slipping. The platform partner could move the infrastructure; nobody on the team had converted a mature SQL Server estate to PostgreSQL before, and nobody could prove the business logic would survive the translation.

The article walks through the method: building the object and dependency inventory first, converting tables, views, stored procedures and functions in dependency order, and running source-to-target reconciliation after every wave rather than once at the end. It covers what converts mechanically, what needs a human (cursors, dynamic SQL, proprietary date handling), and what the reconciliation evidence pack has to contain before an auditor will accept a cutover.

Outcome: 100% of database objects converted, validated and deployed in three months, with audit-ready reconciliation at every step. The AWS migration kicked off on time.

Ask us about this engagement →

RECONCILIATION Data migration reconciliation: a practical checklist for Australian teams Target keywords: data migration validation · source to target reconciliation  ·  Proof: Coal LSL, MetLife

Moving the data is the easy half. The hard half is proving to a business owner, an auditor or a regulator that what arrived is what left.

A working checklist: row-count reconciliation per layer, key-level comparison on the columns that carry business meaning, transformation-rule validation against the specification rather than against the code, exception analysis with a documented disposition for every break, and the evidence capture that turns all of it into something reviewable. Includes the reconciliation report template we hand over at cutover.

Ask us about this engagement →

PRIVACY & COMPLIANCE Automated data retention and disposal under the Privacy Act Target keywords: Privacy Act data retention · data retention and disposal  ·  Proof: The Smith Family (ADDF)

Holding data past its retention period is a compliance exposure that grows quietly. Manual disposal across platforms is slow, error-prone and almost impossible to evidence after the fact.

This covers the Automated Data Disposal and De-identification Framework: how retention rules are expressed as policy rather than scripts, how cascading disposal handles records referenced across systems, how de-identification keeps analytical value in data you are no longer allowed to hold in identified form, and the Power BI reporting that shows leadership what was disposed, what was de-identified and what is due next period.

Ask us about this engagement →

DATA QUALITY Building a SQL-based data quality framework your team can extend Target keywords: automated data quality checks · data quality framework  ·  Proof: MetLife

Most data quality tools are a licence, a UI and a dependency on whoever configured it. Written in SQL instead, the rules live in the warehouse your team already operates, and the engineer who inherits them can read them.

Covers the rule taxonomy (completeness, validity, consistency, timeliness, uniqueness), how to make rules configuration-driven so a new table does not mean new code, where to put the results so they are visible to the business rather than buried in a log, and how the daily failure and remediation report gets read by someone who can act on it.

Ask us about this engagement →

AI DATA READINESS Snowflake Cortex Search over Zendesk and SharePoint without a custom integration layer IN DRAFT Target keywords: Snowflake Cortex Search · enterprise knowledge search  ·  Proof: The Smith Family
PRIVACY & COMPLIANCE PII classification and masking in Snowflake: a step-by-step approach IN DRAFT Target keywords: PII classification · PII data masking  ·  Proof: MetLife
AI DATA READINESS Is your data AI-ready? Ten questions to ask before an AI pilot IN DRAFT Target keywords: AI-ready data · AI readiness assessment  ·  Proof: Founder experience
RECONCILIATION Load audit on Matillion: proving every record arrived IN DRAFT Target keywords: load audit framework · Matillion consultants Australia  ·  Proof: MetLife
DATA ENGINEERING Metadata-driven pipelines: stop rebuilding the same ingestion code IN DRAFT Target keywords: metadata-driven data pipelines · data onboarding automation  ·  Proof: Noviq
DATABASE CONVERSION Oracle to PostgreSQL: what converts automatically and what needs a human IN DRAFT Target keywords: Oracle to PostgreSQL migration Australia · stored procedure conversion  ·  Proof: Noviq Shift
POSITIONING Why your next data project should not start with a new licence IN DRAFT Target keywords: vendor-neutral data consultancy · use existing data platform  ·  Proof: All case studies
PLATFORMS BigQuery vs Snowflake for Australian mid-market teams IN DRAFT Target keywords: BigQuery consultants Australia · Snowflake partner Australia  ·  Proof: Founder certifications

Publishing roughly two a month. Want one of the drafts early? Ask us.

THE EVIDENCE TRAIL

Four artefacts, and who asks for each one.

The deliverable in a data engagement is not the pipeline. It is the ability to answer a reviewer who asks where a number came from. These four artefacts are what makes that answerable, and they come out of every engagement on this page.

WHAT EVERY ENGAGEMENT LEAVES IN YOUR HANDSEVIDENCE 01Reconciliation reportSource-to-target counts and key-levelcomparison, per wave, signed off beforecutover.EVIDENCE 02Data quality baselineRule results per layer with a daily failureand remediation record.EVIDENCE 03Disposal & PII recordWhat was classified, masked, disposed of, andwhen, reportable to a board.EVIDENCE 04The handover packRunbooks, documented lineageand client-owned code, in yourown repositories.
Each artefact answers a different reviewer: an auditor, an analyst, a privacy officer and the team who inherits the build.

So what: ask any firm to show you a redacted version of all four from a previous engagement. If they cannot produce them, those artefacts were never built.

The long-form version

We can walk through the architecture, the reconciliation evidence and the handover pack from any of these four engagements, redacted where the client requires it.

Request a walkthrough Browse the use-case library