14 · Microsoft Fabric · SQL · Python
Data platform and data quality
The report is right until someone finds a duplicate invoice.
Walk the pipeline, correct the batch and publish it again.
Demo on fictional data — never customer data.
Batch of 500 invoices from 14,000 sales invoices. Correcting removes the fault from the publication.
- Raw
- 500
- Rejected
- 21
- Warnings
- 7
- Published
- 479
- 1. Source
- 2. Check for duplicates and required fields
- 3. Conform units
- 4. Report
479 rows can go to the report. 21 stay behind.
SELECT invoice_number, COUNT(*)
FROM invoice_raw
GROUP BY invoice_number
HAVING COUNT(*) > 1;