Why Exceptions Are Crucial in a Modern Data Platform
When we talk about “training” in a data platform, most people think of improving data quality, performance, or business value. But there is something much more fundamental that we need to consistently train: exceptions.
In modern platforms such as Microsoft Fabric, data pipelines and notebooks throw exceptions by default. They issue warnings when something goes wrong. They stop faulty transformations before they continue. They prevent silent data corruption.
Yet this basic rule is often overlooked in custom logic, notebooks, and pipelines. And that’s where things start to go wrong.
When “try… except” becomes “try… accept everything”
In several projects and codebases, I keep coming across the same pattern.
This looks harmless. It seems fine. But in a production data platform, this is the equivalent of sweeping broken glass under the rug. The notebook marks the cell as “successful.” The notebook shuts down “successfully.” The Fabric pipeline shows a green checkmark.
Meanwhile, somewhere, a file failed to load, or an API returned incomplete data. And no one knows about it. Not the engineer. Not the product owner. Not the business. The error was caught, but the truth wasn’t brought to light.
Why Good Exceptions Make Good Platforms
Good data engineering isn’t about suppressing errors. It’s about making them visible early, loudly, and clearly. If you handle exceptions properly, you’ll get:
- Obvious pipeline errors
- Faster debugging
- Accurate Operational Monitoring
- Improved data lineage and root cause analysis
- Stronger governance
- Higher Data Reliability
This is especially important in Fabric, because pipelines and events depend on an accurate runtime status, including errors.
The right strategy: check when necessary, always raise
If you really need to catch an exception for a retry, logging, cleanup, or a custom error message, always rethrow it.
The log is useful. But the raise is necessary. It ensures that your pipeline and all the code above it reflect reality.
Notebooks vs. Pipelines: A Quick Reality Check
Notebooks work correctly until you start using `try`. The pitfall lies in the `except` section. Don’t use it to catch exceptions, but rather to make an adjustment and try again, or to perform additional actions. Always end the `except` block with `raise`. Only in very exceptional cases should you choose not to raise an exception, and only for specific exceptions.
Data Pipelines work correctly as long as you use precedent constraints other than “on-success” (the green lines). “On-completion” (blue), “on-fail” (red), and “on-skipped” (gray) introduce the risk of unintentionally swallowing exceptions. This is because the rule that determines whether a pipeline is reported as failed is based solely on the last executed activity in each branch—the leaf activities. This behavior is explained in the Azure Data Factory documentation and also applies to Fabric pipelines.
A common pattern is to send an email when an error occurs. The pipeline sends the email if the task fails, but the pipeline itself is reported as successful because the leaf activity (the email task) was completed successfully. The same risk applies to “on-completion” and “on-skipped.”
Therefore, use alternatives to “on-success” only when absolutely necessary. And if you do use them, add an explicit “Fail” activity so that the pipeline execution is marked as failed. You can place multiple “Fail” activities in the same pipeline. Even if a “Fail” activity is executed, the pipeline will still process all permitted activities. Only then does the pipeline report a failed run.
The Importance of… Honesty in Code and Pipelines
As the famous play reminds us: being honest from the very beginning prevents many problems later on. The same applies to exceptions.
If you want your data platform to behave predictably, exceptions must bubble up to the orchestration layer, where they belong. When errors are clearly visible, life becomes easier: pipelines behave predictably, teams debug faster, and the platform remains reliable.
Handling exceptions properly isn’t about being strict or dramatic. It’s simply about ensuring that the right systems can respond to what’s actually happening.
Well-handled exceptions make for well-functioning data platforms.
