← all work.

reliability job / in production

Backup Verification

A nightly job that answers one question: if we had to restore from last night's backup, would the data actually be there and readable?

role.

built and own it

status.

in production

type.

reliability job

stack.

python, docker, kubernetes, airflow, azure files, azure backup, microsoft graph

Mockup of the backup verification report: share and VM results, file checks, drift and recovery points
illustrative mockup with sample data. no client data.
3amevery night, run by Airflow on Kubernetes
15+file formats opened with a real parser, not just checked for existence
2halves: file share snapshots and VM recovery points
284automated tests

the problem.

A backup job that says "succeeded" doesn't prove the files inside it are intact or that you could restore them when it matters. You usually find out on the worst possible day.

file shares.

It picks yesterday's Azure Files snapshot, downloads a sample of files and opens each one with a real parser: FoxPro tables and their memo and index files, Excel, CSV, PDF, ZIP, RTF and more. The sample rotates, so over time it covers far more than the same few files. Once a week it fully parses the big files too.

It also watches for what ransomware leaves behind: content that suddenly looks encrypted, ransom note file names, files that disappeared overnight or a large share of a folder rewritten at once. Missing daily snapshots are flagged too.

vm disks.

For each protected VM it confirms there's a recent recovery point and whether a fast restore is still possible or only the slower vault copy is left.

how it runs.

It moved from a dedicated PC under Task Scheduler to a Docker image on Azure Kubernetes Service, triggered by Airflow every night. Results go out as a branded HTML email through Microsoft Graph that works in light and dark mode, with an optional Teams alert that only speaks up when something is wrong.

how it works.

  1. pick yesterday's snapshot
  2. download a rotating sample of files
  3. open each file with a real parser
  4. check for encryption, drift and missing snapshots
  5. check VM recovery points
  6. email the report, alert Teams on failures