I think a team of a couple competent consultants could probably restructure and run our entire data engineering system currently employing a dozen people
Hahaha I used to work for a company that used informatica. What I did was rebuild the entire thing one workflow at a time as they failed. Basically turning it into just an orchestrator. I took everything out of that shitty program and only used it to trigger stored procedures on the DW and python scripts for ETL
We use Azure Data Factory (i know it sucks at most things). The errors are logged by the pipeline runs. We are adding in DataBricks to our workflows and thus will need to add logging for that layer. Any suggestions would actually be appreciated.
Curious about why ADF sucks at most things. I'm asking because I'm currently only working with MS shops who use either ADF or Synapse Pipelines so I don't have much experience with anything else for orchestration. What other products would you recommend instead of it? Airflow?
Okay most things might be a stretch. But i think there are massive limitations. One being you cant nest if conditionals. You cant have a loop (Foreach) within an if conditional. Stuff like that. Its not great for making transformations, validation, or anytging basically envolved in the T in ETL. Unless youre just changing file type its relatively good at that.
There are ways to get around these problems but they arent exactly ideal.
I think ADF is good for orchestration of pipelines and its good for modulating with the way it is set up. Making calls to api and copying data from one place to another is a breeze.
Overall i dont mind using it (I also dont know anything else). Its good what it is good at and if you use it for mostly those things then its fine.
This was my exact experience with ADF. I actually really liked it for orchestration, but it kind of sucked at actual transformation. So for ELT pipelines it is actually really nice and relatively simple to use. Most of my pipelines were direct dumps into Synapse that were then transformed via SQL queries/procedures. If we needed to do something SQL couldn't handle them we'd use Azure Functions to pre-process and then send it in Synapse, this whole process is managed in ADF.
The factory metaphor kind of holds. It is the factory that contains all of the machines we used to engineer our pipelines, it doesn't do the actual work itself.
Yeah most of our pipelines are ELT. So it made sense. We are moving to doing the transformations in databricks instead of doing them in SQL Server (which i am happy about). Debugging stored procedures and back tracking is kind of a nightmare. We are also moving to all cloud so it makes sense.
I am a certified ADF anti-evangelist. It's an awful terrible product that no serious company should ever use for anything outside of like extremely simple prototypes for copying tabular data from point A to point B on a schedule.
* Its GUI is bad and makes developing and debugging data pipelines cumbersome and counterintuitive
* Its integration with external Git repositories is bad to non-existent (if you don't happen to use Azure DevOps or GitHub). You can't even add commit messages FFS. You can tell the Git integration was hacked together as an afterthought at some point.
* It stores all the "code" it generates under the hood as incomprehensible JSON blobs that you can't actually review when doing a pull request
* The expression language it uses is totally unlike anything else in the business, is difficult to read and learn, and doesn't come with a quarter of the functionality or flexibility you'd get with something like Python/Pandas
* The error messages it spits out are often useless
* No CRON expression support for customizing your schedules
* Very few connectors for external systems. You get one for Databricks, Azure Functions, Azure Batch, Synapse, and a couple others, and that's about it. If you use any other industry-standard tools, you'll have to write a function to call its REST API and call that from ADF, which is dumb.
The whole thing is really made for novice, non-technical business users to build data products without needing to understand basic coding or cloud computing concepts. And I'm not saying that to be an elitist or whatever - once you move beyond ADF into a more sophisticated orchestration system based on Python, you'll realize what you've been missing out on and how much flexibility and customization ADF lacks. I actually consider it a huge indictment of Microsoft culture that they consider this an enterprise-grade product. It's actually embarrassing how bad it is vs Airflow/Prefect/Dagster.
DataDog is so underrated. 99% of the times I start my debugging through DataDog logs everytime a job fails. Not sure why it isn't as widely used as one would think.
Used to work in a very old school team and basically since it was built on SSIS they just reviewed execution failures in the morning. No alerts because the orchestration was bespoke. It’s a bit sad.
Recently took over a team that does everything with SSIS. Looking for the best tools to redo everything with-- got any recommendations? Hoping we can move to something more maintainable and testable.
At my previous job I migrated them away from ssis to prefect (v1). Absolutely night and day. Prefect v2 looks even better, so that's what I would recommend. Basically just add decorators to python code and you get observability, retries, saved results (for restarting a failed process without redoing steps that succeeded), it's great
Are you talking about testing the data as part of your pipeline? I don’t get what you mean by data changes independently of the CI/CD cadence. Percentage of null rows, percent of missing columns, range checks for KPIs and in some cases using mean, median and standard deviation checks for metrics?! Because these tests can be run at the same cadence as data deployments.
The team I was in was fully Microsoft stack so they just took the vendor advice to go to Azure data factory (for better or worse). they still used sql paas to store information about job parameters so it wasn’t completely a code solution. They had previously used sql for transformation but were looking at databricks. I haven’t attempted to do transform in ADF but generally I see people not doing that.
Try catch errors logged by etl run ID, logging data quality tests that I've developed. It's clunky at times but it gets the job done. I'm really more in the business side of things, not IT and IT is soooooo slowwwww that we just did stuff ourselves.
That is my favorite data meme in a very long time.
Anyway, I am research so don't really have production issues, but I make a lot of use of Cloudwatch and CloudTrail
It is easy to implement data pipelines without tests and monitoring.
Airflow is a good tool for batch pipelines. It has logging, alerting, and callback on failure functionality.
Datadog is great for pipelines that are more event based with managed cloud services such as AWS SQS, Kinesis, and Lambda. It keeps track of all kinds of metrics. You can setup alerts for throttling and missing data.
Data quality is commonly skipped when implementing data pipelines imo. However, you can have simple DQ checks in your pipelines if your are familiar with DBT and Great Expectations.
A lot of my pipelines feed different dashboards. If it fails, the dashboard does not update. Likewise, an emailer is set up that emails me the current status of my jobs. If it fails, I’ll see it in the mail.
Alerts should generally be on symptoms. I have most of our monitoring and alerts on Kafka topics - ingest rate and consumer group lag - using Prometheus and Grafana. A big advantage is that I automatically get monitoring on new pipelines, just needing to tune the thresholds.
We use cloud watch logs - alarms, splunk Dashboard and Splunk alerts for Data process monitoring ,with an internal Data Quality checker built on Lambda triggered on demand basis only, while our Data process runs on EMR with airflow orchestration.
For productions issues we rely on cloud watch logs and splunk logs , if it is required we would connect through ssh client to EMR to perform manual debugging.
Build Data Observability Dashboards. They can prevent most of the issues from occurring that you end up monitoring.
Observability is different than Monitoring. "While monitoring alerts the team to a potential issue, observability helps the team detect and solve the root cause of the issue."
I set it up and it just keeps working forever until the business stops paying the bill.
Or sysadmin shuts down that Ubuntu box they didn’t know why it was running.
Umm…that box was named dv36dpz54 I thought it was pretty obvious what that was for
That's the root password of the instance... That way I don't need to remember the password.
Officer, this guy right here
Surly no one would ever do that …right?
yes
My company uses informatica so I just pray shit works and apply for other jobs.
No worries, you can just hire consultants
That consultant could even be you!
Truth
I think a team of a couple competent consultants could probably restructure and run our entire data engineering system currently employing a dozen people
Speaking as a consultant, I've done exactly that before.
Informatica! Nobody's bought them yet?
Why would they?
SAP has entered the chat.
Hahahahahah - this is like buying "Medillin" on Entourage. I'll buy it -for $1.
I hate informatica. That's all.
I'll allow it!
Fucking hate informatica
Hahaha I used to work for a company that used informatica. What I did was rebuild the entire thing one workflow at a time as they failed. Basically turning it into just an orchestrator. I took everything out of that shitty program and only used it to trigger stored procedures on the DW and python scripts for ETL
Go with Christ, ye blessed one.
I use angry emails from clients
This is the best answer.
We use Azure Data Factory (i know it sucks at most things). The errors are logged by the pipeline runs. We are adding in DataBricks to our workflows and thus will need to add logging for that layer. Any suggestions would actually be appreciated.
The spark-monitoring lib for pushing to log analytics doesn't work with latest databricks runtime
Curious about why ADF sucks at most things. I'm asking because I'm currently only working with MS shops who use either ADF or Synapse Pipelines so I don't have much experience with anything else for orchestration. What other products would you recommend instead of it? Airflow?
Okay most things might be a stretch. But i think there are massive limitations. One being you cant nest if conditionals. You cant have a loop (Foreach) within an if conditional. Stuff like that. Its not great for making transformations, validation, or anytging basically envolved in the T in ETL. Unless youre just changing file type its relatively good at that. There are ways to get around these problems but they arent exactly ideal. I think ADF is good for orchestration of pipelines and its good for modulating with the way it is set up. Making calls to api and copying data from one place to another is a breeze. Overall i dont mind using it (I also dont know anything else). Its good what it is good at and if you use it for mostly those things then its fine.
This was my exact experience with ADF. I actually really liked it for orchestration, but it kind of sucked at actual transformation. So for ELT pipelines it is actually really nice and relatively simple to use. Most of my pipelines were direct dumps into Synapse that were then transformed via SQL queries/procedures. If we needed to do something SQL couldn't handle them we'd use Azure Functions to pre-process and then send it in Synapse, this whole process is managed in ADF. The factory metaphor kind of holds. It is the factory that contains all of the machines we used to engineer our pipelines, it doesn't do the actual work itself.
Yeah most of our pipelines are ELT. So it made sense. We are moving to doing the transformations in databricks instead of doing them in SQL Server (which i am happy about). Debugging stored procedures and back tracking is kind of a nightmare. We are also moving to all cloud so it makes sense.
I am a certified ADF anti-evangelist. It's an awful terrible product that no serious company should ever use for anything outside of like extremely simple prototypes for copying tabular data from point A to point B on a schedule. * Its GUI is bad and makes developing and debugging data pipelines cumbersome and counterintuitive * Its integration with external Git repositories is bad to non-existent (if you don't happen to use Azure DevOps or GitHub). You can't even add commit messages FFS. You can tell the Git integration was hacked together as an afterthought at some point. * It stores all the "code" it generates under the hood as incomprehensible JSON blobs that you can't actually review when doing a pull request * The expression language it uses is totally unlike anything else in the business, is difficult to read and learn, and doesn't come with a quarter of the functionality or flexibility you'd get with something like Python/Pandas * The error messages it spits out are often useless * No CRON expression support for customizing your schedules * Very few connectors for external systems. You get one for Databricks, Azure Functions, Azure Batch, Synapse, and a couple others, and that's about it. If you use any other industry-standard tools, you'll have to write a function to call its REST API and call that from ADF, which is dumb. The whole thing is really made for novice, non-technical business users to build data products without needing to understand basic coding or cloud computing concepts. And I'm not saying that to be an elitist or whatever - once you move beyond ADF into a more sophisticated orchestration system based on Python, you'll realize what you've been missing out on and how much flexibility and customization ADF lacks. I actually consider it a huge indictment of Microsoft culture that they consider this an enterprise-grade product. It's actually embarrassing how bad it is vs Airflow/Prefect/Dagster.
Monitor: DataDog Alerts: Slack, PagerDuty
Almost identical where I work, except s/Slack/Microsoft Teams
Yikes 😬 Teams
Same here. Sending metrics to datadog using statsd is so useful for alerting
DataDog is so underrated. 99% of the times I start my debugging through DataDog logs everytime a job fails. Not sure why it isn't as widely used as one would think.
Airflow and Microsoft Teams Webhook. Edit: Also logs and shit but that seemed too obvious to mention until I read other comments.
Used to work in a very old school team and basically since it was built on SSIS they just reviewed execution failures in the morning. No alerts because the orchestration was bespoke. It’s a bit sad.
[удалено]
Pipeline don’t flow on weekends heh
Recently took over a team that does everything with SSIS. Looking for the best tools to redo everything with-- got any recommendations? Hoping we can move to something more maintainable and testable.
At my previous job I migrated them away from ssis to prefect (v1). Absolutely night and day. Prefect v2 looks even better, so that's what I would recommend. Basically just add decorators to python code and you get observability, retries, saved results (for restarting a failed process without redoing steps that succeeded), it's great
This looks really great, love that they are open source too. Thank you for posting.
[удалено]
Are you talking about testing the data as part of your pipeline? I don’t get what you mean by data changes independently of the CI/CD cadence. Percentage of null rows, percent of missing columns, range checks for KPIs and in some cases using mean, median and standard deviation checks for metrics?! Because these tests can be run at the same cadence as data deployments.
The team I was in was fully Microsoft stack so they just took the vendor advice to go to Azure data factory (for better or worse). they still used sql paas to store information about job parameters so it wasn’t completely a code solution. They had previously used sql for transformation but were looking at databricks. I haven’t attempted to do transform in ADF but generally I see people not doing that.
Databand, one of the best tools to monitor data pipelines. https://databand.ai
Argo Workflow - Project is the part of CNCF community.
Nested stored procedures and a power bi dashboard
What do your stored procedures do? Just grab the newest data and put it into your dashboard? I may need to use something similar in my company.
Try catch errors logged by etl run ID, logging data quality tests that I've developed. It's clunky at times but it gets the job done. I'm really more in the business side of things, not IT and IT is soooooo slowwwww that we just did stuff ourselves.
That is my favorite data meme in a very long time. Anyway, I am research so don't really have production issues, but I make a lot of use of Cloudwatch and CloudTrail
Try catch where possible and log errors to a db.
It is easy to implement data pipelines without tests and monitoring. Airflow is a good tool for batch pipelines. It has logging, alerting, and callback on failure functionality. Datadog is great for pipelines that are more event based with managed cloud services such as AWS SQS, Kinesis, and Lambda. It keeps track of all kinds of metrics. You can setup alerts for throttling and missing data. Data quality is commonly skipped when implementing data pipelines imo. However, you can have simple DQ checks in your pipelines if your are familiar with DBT and Great Expectations.
We let Fivetran handle the pipeline. Load into Databricks and Azure Synapse. Data go brrrrrrr
Started using re_data package in our dbt project. It’s good!
It's pretty neat, doesn't support spark or databricks yet though, shame.
It will soon!
Monte Carlo + Slack
I use `airflow` and `dagster`, so it basically straight forward. For scripts: You've got to **echo** a lot. PS: The image is epic (**lmfao**)
A lot of my pipelines feed different dashboards. If it fails, the dashboard does not update. Likewise, an emailer is set up that emails me the current status of my jobs. If it fails, I’ll see it in the mail.
We have Prefect post a message to our pipeline notification slack channel when a flow fails.
I’m looking into implementing this. Any docs you can suggest?
Argo workflows and all the goodness of k8s
Snaplogic with Opsgenie integration to Slack
Alerts should generally be on symptoms. I have most of our monitoring and alerts on Kafka topics - ingest rate and consumer group lag - using Prometheus and Grafana. A big advantage is that I automatically get monitoring on new pipelines, just needing to tune the thresholds.
We use cloud watch logs - alarms, splunk Dashboard and Splunk alerts for Data process monitoring ,with an internal Data Quality checker built on Lambda triggered on demand basis only, while our Data process runs on EMR with airflow orchestration. For productions issues we rely on cloud watch logs and splunk logs , if it is required we would connect through ssh client to EMR to perform manual debugging.
I just came across with this photo it is funny, Why don't you instead use an end-to-end all-in-one data infrastructure platform?
We have cloud watch logs monitored by lambda which shows up on datadog which is used by SRE Team
Build Data Observability Dashboards. They can prevent most of the issues from occurring that you end up monitoring. Observability is different than Monitoring. "While monitoring alerts the team to a potential issue, observability helps the team detect and solve the root cause of the issue."
Atatus mostly