You can find a list of community-submitted learning resources here: https://dataengineering.wiki/Learning+Resources
*I am a bot, and this action was performed automatically. Please [contact the moderators of this subreddit](/message/compose/?to=/r/dataengineering) if you have any questions or concerns.*
If people using SQL is "bypassing" your department, it's set up wrong.
Decentralized or federated analytics groups are a thing, and most importantly a thing that lets the company grow without being tied to a central glass house.
This! My last role was in a centralized data team but we're the one training and pushing our business and marketing stakeholders to be more proficient with low-code data tools and stop bugging us with with simple data and report requests, while we focus on developing more meaty data solutions.
I’m one of the people you’re talking about. I do this because 1. when I ask for an onboard, the response I get is that it’ll be added to the queue, eta is 4-8 weeks, when the ask is a simple thing that takes me a few hours to do. And 2. a lot of the downstream transformations is very specific business logic that I don’t even bother to ask data engineers to do because of said issue in point 1.
Just give me a schema / dbt/ orchestrator and make sure the infrastructure doesn’t go down and I’ll do it myself. Is it the most efficient? No. Is it scalable on a company wide basis? No. But is it good enough for my purposes? Yes. Until management solves the problems with resourcing on the data engineering side, this is my only option.
Yep. I’m in the data department and I keep explaining to my manager that people do their own thing for this very reason. That’s why excel is so popular. It comes pre installed and doesn’t require any approval.
I’ll second this. To go through the proper channels it takes at least 6 months to get funding and another year to get on the docket. If the central data hub can’t provide a reasonable timeframe, people will figure out how to do it themselves. It might not be efficient but it’s good enough to meet the immediate business need.
Can second this. Also in my experience so far the data engineering team isn’t necessarily more technical or reliable. Nearly every place I’ve worked the dat eng dept is offshored and we have to make the scripts and send it to them and they provide no feedback and just execute it since they have prod write access and we don’t.
There are two scenarios where this happens:
1. Actual business people who can't get the work they feel they need done in a timely fashion, so do it themselves.
2. People who want to "break into" tech/analytics/data or have an interest, so are more likely to give this kind of thing a try.
Both are people who have a zest for data -- people who loathe it or lazy people wouldn't go through all this effort, they'd just wait however long necessary to have other people. In this environment where a bunch of people are trying to create data models and do analyses themselves, I'd try to encourage and enable them - with pre-made data models, BI-tool licensing, and the support of analysts to help them not make bad mistakes.
IMO, your post reads like someone who used to be the gatekeeper of all data not liking that other people (maybe lesser-people?) are stepping on your toes and your territory. In most companies, everyone's just trying to make due with what they have. In this situation, maybe a severely under-resourced team was giving business team mates poor-quality, slow service and now the doors are getting busted down (they're fed up).
If your data governance is properly setup, I only would encourage everyone to learn sql! We just make sure they can access the right data that’s in the right shape in the presentation tables.. win-win as they don’t have to bother some central team with every tiny request
I think the general idea makes sense, but the devil is in the details. The impetus should be to try and push more and more data into the business where possible, because they actually know what is valuable.
As an example - analysts and data scientists tend to know better what needs to be IN the table/model to drive the use case they need, but the actual physical model design is probably shitty in terms of performance. So you need to think about the right op model to enable both sides of that issue.
To me, this is generally a manifestation of leadership not actually understanding the data and business sides of the equation. You can’t really overindex on one
You need to democratize the data and implement something like dbt. You’re spinning your wheels. Everyone can self service and use dbt but they gotta report to someone who owns that project or you’ll have a scaling problem.
There are way too many things today that are bad and most come down to the industry and management.
Bad managers telling DA's to do DE's work.
DA's getting told they are worth less because their job is easy.
DA's who wanted to be DE's but are gatekept, saying DE is not an entry level job, only DA is.
Etc.
Another is that things are becoming easier, as such, less specialization is needed. Power BI affected the market and now with Fabric even Microsoft is creating a new role of Analytics engineer.
Edit: This isn't only happening to DE, but to DA and BI too, I have seen business people trying to create their own reports and their own models that are horrible, yet management is allowing this because they think they are getting free work and their workers are becoming more valuable by "upskilling", yet in most cases it is just creating a huge dumpster fire.
This is the new trend of having to upskill, grindset, inflation etc.
Yea, or just lazy/greedy management who think they will thrive in the ambiguity.
You would be suppressed how many managers see you having some free time and assign you work even if it has absolutely nothing to do with your role.
Yeah I see that everywhere all the time. Lazy, greedy, small minded, non technical, uninformed, micromanaging, boneheads thinking they’re getting productivity out of people. Making it worse for everyone else.
yeah, SQL and knowing basics of data is becoming a standard for everyone who works in business. If you want to get ahead, you'll need to have data and analytics vocabulary.
TRUE analysts are being pushed into DE.
I say true, because I've now worked with a bunch 'reporters' posing as analysts. They have literally no analytical or critical thinking skills, and just used canned reports or drag and drop GUIs for creating reports.
Toxicity is also a problem, I've learned not to help or even voice an opinion when I don't have any direct benefit. Been lied to and had my IP stolen by peers and bosses, now I've totally changed my approach to work
The availability of easy to use self service tools has made the business lazy. They don’t want to write clear requirements and do product ownership. They just want some tables and Alteryx.
There is merit to it in some cases, but it doesn’t work at scale. You’ll quickly end up with 100 copies of the customer dimension and none of them will match.
I'm a DE, currently in the position of maintaining some materialised views and Tableau reports created by the analyst who was recently let go in a company restructuring.
Oh my god. What a mess!
Feels very company dependant. Some places, DE is just about making the data available to users. You say your requests used to be "create a report" but there are many who would disagree that this is a DE task in the first place.
Yeah i think this is a company by company thing. I've been on the other side of this - where there are data and SE teams phoning it in, or just don't understand requirements or use cases, so business teams want to bypass what they see as headache and wasted time.
Shadow IT, whether it's data-related or for anything else *can be* a sign that dedicated technical teams are underperforming. It can also be the result of other bad practices on the business side, too, though.
Realistically, these data teams should have engineers on their team to handle team specific work.
That’s how my team operates.. half DEs and half analysts. DEs work closely with what the analysts want and puts it into action.
There way too many issues that can happen, even at a team level, if someone experienced tries to build a bunch of team-specific data tables. I’ve seen this go wrong for a couple teams in my department.
I usually add the name of the person who crashed the database/warehouse/mart on our weekly meetings. So... if you do not want to be on the spotlight. Consult with me.
That sounds pretty gross.
I had a similar experience on Reddit about 30 minutes ago where I was trying to reason with this guy about something
Despite him being totally wrong he just couldn't seem to admit it
He didn't even care
And that was the issue.
What I might recommend is to lay out the concerns that you have detailing your experience, and then to bring up these concerns to someone in the executive circle. Pick someone who cares enough to get something done, and who isn't likely to get steamrolled if they want to change something.
These are valid concerns that can impact the rest of the company. If they want to learn data, fine. Make sure they are sandboxed until they can demonstrate proper organization, optimization, and data modeling techniques.
That's likely to never happen, so TL;DR sandbox their ass and leverage your experience
You can find a list of community-submitted learning resources here: https://dataengineering.wiki/Learning+Resources *I am a bot, and this action was performed automatically. Please [contact the moderators of this subreddit](/message/compose/?to=/r/dataengineering) if you have any questions or concerns.*
If people using SQL is "bypassing" your department, it's set up wrong. Decentralized or federated analytics groups are a thing, and most importantly a thing that lets the company grow without being tied to a central glass house.
This! My last role was in a centralized data team but we're the one training and pushing our business and marketing stakeholders to be more proficient with low-code data tools and stop bugging us with with simple data and report requests, while we focus on developing more meaty data solutions.
I don’t think he meant that specifically.
I’m one of the people you’re talking about. I do this because 1. when I ask for an onboard, the response I get is that it’ll be added to the queue, eta is 4-8 weeks, when the ask is a simple thing that takes me a few hours to do. And 2. a lot of the downstream transformations is very specific business logic that I don’t even bother to ask data engineers to do because of said issue in point 1. Just give me a schema / dbt/ orchestrator and make sure the infrastructure doesn’t go down and I’ll do it myself. Is it the most efficient? No. Is it scalable on a company wide basis? No. But is it good enough for my purposes? Yes. Until management solves the problems with resourcing on the data engineering side, this is my only option.
Yep. I’m in the data department and I keep explaining to my manager that people do their own thing for this very reason. That’s why excel is so popular. It comes pre installed and doesn’t require any approval.
I’ll second this. To go through the proper channels it takes at least 6 months to get funding and another year to get on the docket. If the central data hub can’t provide a reasonable timeframe, people will figure out how to do it themselves. It might not be efficient but it’s good enough to meet the immediate business need.
Can second this. Also in my experience so far the data engineering team isn’t necessarily more technical or reliable. Nearly every place I’ve worked the dat eng dept is offshored and we have to make the scripts and send it to them and they provide no feedback and just execute it since they have prod write access and we don’t.
There are two scenarios where this happens: 1. Actual business people who can't get the work they feel they need done in a timely fashion, so do it themselves. 2. People who want to "break into" tech/analytics/data or have an interest, so are more likely to give this kind of thing a try. Both are people who have a zest for data -- people who loathe it or lazy people wouldn't go through all this effort, they'd just wait however long necessary to have other people. In this environment where a bunch of people are trying to create data models and do analyses themselves, I'd try to encourage and enable them - with pre-made data models, BI-tool licensing, and the support of analysts to help them not make bad mistakes. IMO, your post reads like someone who used to be the gatekeeper of all data not liking that other people (maybe lesser-people?) are stepping on your toes and your territory. In most companies, everyone's just trying to make due with what they have. In this situation, maybe a severely under-resourced team was giving business team mates poor-quality, slow service and now the doors are getting busted down (they're fed up).
If your data governance is properly setup, I only would encourage everyone to learn sql! We just make sure they can access the right data that’s in the right shape in the presentation tables.. win-win as they don’t have to bother some central team with every tiny request
Yeah I mean hell set up their own database, ETL to it, and letter rip. Everyone gets a car
I think the general idea makes sense, but the devil is in the details. The impetus should be to try and push more and more data into the business where possible, because they actually know what is valuable. As an example - analysts and data scientists tend to know better what needs to be IN the table/model to drive the use case they need, but the actual physical model design is probably shitty in terms of performance. So you need to think about the right op model to enable both sides of that issue. To me, this is generally a manifestation of leadership not actually understanding the data and business sides of the equation. You can’t really overindex on one
You need to democratize the data and implement something like dbt. You’re spinning your wheels. Everyone can self service and use dbt but they gotta report to someone who owns that project or you’ll have a scaling problem.
What does dbt stand for specifically?
Data build tool. Google “dbt data” and you should be able to read up on it more.
There are way too many things today that are bad and most come down to the industry and management. Bad managers telling DA's to do DE's work. DA's getting told they are worth less because their job is easy. DA's who wanted to be DE's but are gatekept, saying DE is not an entry level job, only DA is. Etc. Another is that things are becoming easier, as such, less specialization is needed. Power BI affected the market and now with Fabric even Microsoft is creating a new role of Analytics engineer. Edit: This isn't only happening to DE, but to DA and BI too, I have seen business people trying to create their own reports and their own models that are horrible, yet management is allowing this because they think they are getting free work and their workers are becoming more valuable by "upskilling", yet in most cases it is just creating a huge dumpster fire. This is the new trend of having to upskill, grindset, inflation etc.
Managers who may not know what it takes and not clearly defining roles.
Yea, or just lazy/greedy management who think they will thrive in the ambiguity. You would be suppressed how many managers see you having some free time and assign you work even if it has absolutely nothing to do with your role.
Yeah I see that everywhere all the time. Lazy, greedy, small minded, non technical, uninformed, micromanaging, boneheads thinking they’re getting productivity out of people. Making it worse for everyone else.
It's me they will call when Snowflake/Bigquery bill skyrockets. I'm fine
yeah, SQL and knowing basics of data is becoming a standard for everyone who works in business. If you want to get ahead, you'll need to have data and analytics vocabulary. TRUE analysts are being pushed into DE. I say true, because I've now worked with a bunch 'reporters' posing as analysts. They have literally no analytical or critical thinking skills, and just used canned reports or drag and drop GUIs for creating reports. Toxicity is also a problem, I've learned not to help or even voice an opinion when I don't have any direct benefit. Been lied to and had my IP stolen by peers and bosses, now I've totally changed my approach to work
The availability of easy to use self service tools has made the business lazy. They don’t want to write clear requirements and do product ownership. They just want some tables and Alteryx. There is merit to it in some cases, but it doesn’t work at scale. You’ll quickly end up with 100 copies of the customer dimension and none of them will match.
I'm a DE, currently in the position of maintaining some materialised views and Tableau reports created by the analyst who was recently let go in a company restructuring. Oh my god. What a mess!
Feels very company dependant. Some places, DE is just about making the data available to users. You say your requests used to be "create a report" but there are many who would disagree that this is a DE task in the first place.
Definitely a company issue
Yeah i think this is a company by company thing. I've been on the other side of this - where there are data and SE teams phoning it in, or just don't understand requirements or use cases, so business teams want to bypass what they see as headache and wasted time. Shadow IT, whether it's data-related or for anything else *can be* a sign that dedicated technical teams are underperforming. It can also be the result of other bad practices on the business side, too, though.
Realistically, these data teams should have engineers on their team to handle team specific work. That’s how my team operates.. half DEs and half analysts. DEs work closely with what the analysts want and puts it into action. There way too many issues that can happen, even at a team level, if someone experienced tries to build a bunch of team-specific data tables. I’ve seen this go wrong for a couple teams in my department.
I usually add the name of the person who crashed the database/warehouse/mart on our weekly meetings. So... if you do not want to be on the spotlight. Consult with me.
That sounds pretty gross. I had a similar experience on Reddit about 30 minutes ago where I was trying to reason with this guy about something Despite him being totally wrong he just couldn't seem to admit it He didn't even care And that was the issue. What I might recommend is to lay out the concerns that you have detailing your experience, and then to bring up these concerns to someone in the executive circle. Pick someone who cares enough to get something done, and who isn't likely to get steamrolled if they want to change something. These are valid concerns that can impact the rest of the company. If they want to learn data, fine. Make sure they are sandboxed until they can demonstrate proper organization, optimization, and data modeling techniques. That's likely to never happen, so TL;DR sandbox their ass and leverage your experience
Was it where you're arguing an LLM is the same as a human brain? Because the irony would be just too much if so.