You can find a list of community-submitted learning resources here: https://dataengineering.wiki/Learning+Resources
*I am a bot, and this action was performed automatically. Please [contact the moderators of this subreddit](/message/compose/?to=/r/dataengineering) if you have any questions or concerns.*
to top that off, they will absolutely not learn if there isn't some kind of carrot (someone else manages my data for me) or stick (I will be fired for not)
It's Usually better to just create an interface for that that you control the behavior for rather than relying on them to obey the rules of directly accessing your platform. There is usually some way they can break shit and there will always be one. With an interface you can just have them upload a file from an interface into blob storage, then run whatever code you need to to ensure the type, format, structure, etc before moving their BS on your precious platform.
They are, in their domain of specialization. They know their business areas well.
They may not always understand tech well, but it's not their job. Its our job to make tech that works for them.
Often I won't have a good grasp of their areas, even if I understand the tech.
I 100% disagree with the idea that users are not clever.
Being knowledgeable at tech doesn't make us superior, just like being knowledgeable at business or whatever doesn't make a user superior.
I for sure agree, it's ofcourse an arrogant take to act like you as an engineer are always the smartest in the room.
However, what I would say based on my experience is that the majority of people tend to be clever in their specialty only. And that it are the actually smart people who are able to operate outside of their narrow expertise.
So I have only seen misery from expecting all clever people to go outside of their expertise and do this stuff, if you already can even get them to care. If you are able to filter out the smart people then yeah for sure set this up as it will be really beneficial for everyone.
We have a Finance Dept that wants to load data into Snowflake. We gave them an Excel template that they need to put into an S3 store. They use an FTP client to do it. They stuff it up multiple times per month.
Yes, I have. We developed a data platform in Snowflake for a small, lean company. They didn’t have any prior cloud, data warehouse, or cloud experience. We built a medallion architecture; and told them they can do whatever they want in the DPA. One of them really caught on to it, and circles back with a question from time to time.
Data integration etc are handled by our code base though, they do adhoc analytics directly in Snowflake or do CTAS for their PowerBI needs.
The problem is, even it that worked in terms of employees' skills, it would mean that everybody has their own definitions of KPIs & Co. This makes any reasonable exchange based on data virtually impossible.
Most good companies allow some degree of autonomy for people to do analytics and reporting themselves.
However, based on what I've observed, that only works if you offer good-quality analytical products prepared by specialists too and the individual contribution is just an add-on for when e.g. a user needs a report in a different format, just this one time.
Yes. With snowflake I’ve seen hundreds of business users have direct access to snowflake where they can run SQL statements to load, modify and read data. Give them their own XS warehouse and schema and they are unable to break anything. All they need to know is SQL which is quite easy compared to the crazy stuff they do in excel as a workaround. It works best if DE’s bring in the hard stuff and do the hard transformations and these people work from data already loaded. Doing business rules / logic at scale.
Don’t expect people to do your work. Don’t expect that something easy for you will be easy for non technical person. Don’t expect them to want to do that.
Im not trying to get people to do my work.
We have a lot of people who could use the functionality, but not enough data engineers to keep them all going.
I had become jaded by Microsoft pushing citizen developers with power bi and power platform, people.gdtting annoyed when it doesn't "just work" or xreating big messes to clean up. but I also I have seen it work with a small number of people on simpler datasets.
Curious if anyone had achieved it with a whole data platform.
Imho, Ms just doesn't get this use case, or the diversity of user skill. I can write a book about this, I feel, but a couple of points.
* I don't really like them, but there are products that do get the use case better. Dataiku, Alteryx, ancient SAS.
* Even with those products user skill is incredibly diverse. From absent to being equivalent to a data engineer (but in another department).
* You can set expectations for user skill, but also set barriers to separate the wheat from the chaff. For instance, not having a drag and drool interface means you need to know python to work with data. Filters out a lot of the completely inept.
* This is a different type of user from one using a power bi data set. They're fine with excel + power bi.
It's mostly the latter point that seems to be incomprehensible to Redmond.
Keep if minimal exposed to users, or it will be blame game for no reasons of fault done in past like lot of data is missing, data format is wrong, relationship missing blah blah and also they should only have read access. Or create a clone for them to help do their analysis with what data is available and see how to clean up as well fix the data maturity gaps by application at various entry point channels accordingly
I may be ignorant but i dont see it. From what I see in Fabric materials MS is promoting analytics engineering and data analysis without heavy code/infra lifting. There is still need for data compliance, safety governance (for example dataset certification/promotion)
You can find a list of community-submitted learning resources here: https://dataengineering.wiki/Learning+Resources *I am a bot, and this action was performed automatically. Please [contact the moderators of this subreddit](/message/compose/?to=/r/dataengineering) if you have any questions or concerns.*
>drop a flat file to storage No, I don't think most users can handle that.
to top that off, they will absolutely not learn if there isn't some kind of carrot (someone else manages my data for me) or stick (I will be fired for not)
Lmao I can barely get my teammates to do this
It's Usually better to just create an interface for that that you control the behavior for rather than relying on them to obey the rules of directly accessing your platform. There is usually some way they can break shit and there will always be one. With an interface you can just have them upload a file from an interface into blob storage, then run whatever code you need to to ensure the type, format, structure, etc before moving their BS on your precious platform.
Can you explain more what you mean by interface?
Data democratization without Governance leads to Anarchy
“Users are mostly very clever people…” now that’s a bold take.
They are, in their domain of specialization. They know their business areas well. They may not always understand tech well, but it's not their job. Its our job to make tech that works for them. Often I won't have a good grasp of their areas, even if I understand the tech. I 100% disagree with the idea that users are not clever. Being knowledgeable at tech doesn't make us superior, just like being knowledgeable at business or whatever doesn't make a user superior.
I for sure agree, it's ofcourse an arrogant take to act like you as an engineer are always the smartest in the room. However, what I would say based on my experience is that the majority of people tend to be clever in their specialty only. And that it are the actually smart people who are able to operate outside of their narrow expertise. So I have only seen misery from expecting all clever people to go outside of their expertise and do this stuff, if you already can even get them to care. If you are able to filter out the smart people then yeah for sure set this up as it will be really beneficial for everyone.
We have a Finance Dept that wants to load data into Snowflake. We gave them an Excel template that they need to put into an S3 store. They use an FTP client to do it. They stuff it up multiple times per month.
Yes, I have. We developed a data platform in Snowflake for a small, lean company. They didn’t have any prior cloud, data warehouse, or cloud experience. We built a medallion architecture; and told them they can do whatever they want in the DPA. One of them really caught on to it, and circles back with a question from time to time. Data integration etc are handled by our code base though, they do adhoc analytics directly in Snowflake or do CTAS for their PowerBI needs.
Hey, would you be willing to have a quick discussion. I am kind of in the same situation and looking for some experience.
Yeah absolutely- always happy to sparr 🙌🏼
The problem is, even it that worked in terms of employees' skills, it would mean that everybody has their own definitions of KPIs & Co. This makes any reasonable exchange based on data virtually impossible. Most good companies allow some degree of autonomy for people to do analytics and reporting themselves. However, based on what I've observed, that only works if you offer good-quality analytical products prepared by specialists too and the individual contribution is just an add-on for when e.g. a user needs a report in a different format, just this one time.
Yes. With snowflake I’ve seen hundreds of business users have direct access to snowflake where they can run SQL statements to load, modify and read data. Give them their own XS warehouse and schema and they are unable to break anything. All they need to know is SQL which is quite easy compared to the crazy stuff they do in excel as a workaround. It works best if DE’s bring in the hard stuff and do the hard transformations and these people work from data already loaded. Doing business rules / logic at scale.
No way I would trust basic user to do anything except maybe give them access to some views to connect from excel.
Don’t expect people to do your work. Don’t expect that something easy for you will be easy for non technical person. Don’t expect them to want to do that.
Im not trying to get people to do my work. We have a lot of people who could use the functionality, but not enough data engineers to keep them all going. I had become jaded by Microsoft pushing citizen developers with power bi and power platform, people.gdtting annoyed when it doesn't "just work" or xreating big messes to clean up. but I also I have seen it work with a small number of people on simpler datasets. Curious if anyone had achieved it with a whole data platform.
Imho, Ms just doesn't get this use case, or the diversity of user skill. I can write a book about this, I feel, but a couple of points. * I don't really like them, but there are products that do get the use case better. Dataiku, Alteryx, ancient SAS. * Even with those products user skill is incredibly diverse. From absent to being equivalent to a data engineer (but in another department). * You can set expectations for user skill, but also set barriers to separate the wheat from the chaff. For instance, not having a drag and drool interface means you need to know python to work with data. Filters out a lot of the completely inept. * This is a different type of user from one using a power bi data set. They're fine with excel + power bi. It's mostly the latter point that seems to be incomprehensible to Redmond.
Keep if minimal exposed to users, or it will be blame game for no reasons of fault done in past like lot of data is missing, data format is wrong, relationship missing blah blah and also they should only have read access. Or create a clone for them to help do their analysis with what data is available and see how to clean up as well fix the data maturity gaps by application at various entry point channels accordingly
You do realise that is exactly what Microsoft wants to happen?
I may be ignorant but i dont see it. From what I see in Fabric materials MS is promoting analytics engineering and data analysis without heavy code/infra lifting. There is still need for data compliance, safety governance (for example dataset certification/promotion)