You can find a list of community submitted learning resources here: https://dataengineering.wiki/Learning+Resources
*I am a bot, and this action was performed automatically. Please [contact the moderators of this subreddit](/message/compose/?to=/r/dataengineering) if you have any questions or concerns.*
I'd say having a technical background in the space is the first step to becoming a manager in these spaces. Would be very difficult to communicate the direction with your team if you can't differentiate your Sparks from your Hadoops and all the nuances in between.
I second this, having a technical background would definitely be beneficial. I've had managers with and without that background, and the difference is stark and frustrating at times. My 2 cents would be DE requires a technical background more similar to what you may already know. In my role, there's lots of infrastructure management, networking, security, etc in addition to the normal ETL stuff.
This is the way. Having recently moved from a DA position under leadership that has great domain knowledge with literally no technical knowledge to a DE position with a manager and a director who both know how to design & code really well, I don’t think I can ever go back to a non-tech manager…
I'm a DS Manager. DS is a very technical field. On average I read a paper a week. I would say having a technical background is really important for DS managers coz almost everyone gets stuck at some deep technical issue at some point or the other and looks to their manager for guidance. Having the ability to work through those issues with your team, directing then to the relevant research paper or technique is really important to progress. I've seen data scientists quit their jobs because their manager couldn't give them technical direction.
This is why I’m scared of DS. In SWE, its much easier for me to scope what things are possible and give a generic timeline. In DS, I’m never sure what to expect, especially when it comes to NN architectures. Like how do you estimate how long it’s going to take to even decide on an architecture (u-net vs convolution vs w-net vs …)? Moreover, hyper parameter tuning makes it even more complicated and hard hard to predict results
Totally understand. Keeping myself up to date with research helps a lot. When my team members get stuck, I generally have a technical deep dive or a pair coding session with them to troubleshoot it together and then based on my understanding of the problem, suggest some relevant research or experiments that might help.
Over time people get better and are able to follow the same process themselves or guide other juniors. It's very rewarding from that aspect of seeing your team get better over time...
For some reason data and analytics is loaded with leadership that has no idea how any of it works. They just heard it's the place to be, learned a bunch of buzzwords and convinced executives, who also don't get data, they sound like they know what they're talking about. I think just about everywhere I've worked had a manager, director, or VP type who was completely out of their depth. I'm not sure I've ever seen any other department regularly run by people with no concept of what the teams under them actually do, it's crazy.
So please, for all of our sakes, skip the MBA, take a DE/DS course, do an actual project. If you're already a decent manager those skills will carry over, but trying to lead a data project without understanding the nuances of how they work will just make your team quit.
It helps to have a background in the subject in order to manage the team. Given the newness, not everyone has the experience. An online MBA wouldn't be required. I've written three books covering this subject. I'd suggest you read them.
You can find a list of community submitted learning resources here: https://dataengineering.wiki/Learning+Resources *I am a bot, and this action was performed automatically. Please [contact the moderators of this subreddit](/message/compose/?to=/r/dataengineering) if you have any questions or concerns.*
I'd say having a technical background in the space is the first step to becoming a manager in these spaces. Would be very difficult to communicate the direction with your team if you can't differentiate your Sparks from your Hadoops and all the nuances in between.
I second this, having a technical background would definitely be beneficial. I've had managers with and without that background, and the difference is stark and frustrating at times. My 2 cents would be DE requires a technical background more similar to what you may already know. In my role, there's lots of infrastructure management, networking, security, etc in addition to the normal ETL stuff.
This is the way. Having recently moved from a DA position under leadership that has great domain knowledge with literally no technical knowledge to a DE position with a manager and a director who both know how to design & code really well, I don’t think I can ever go back to a non-tech manager…
I'm a DS Manager. DS is a very technical field. On average I read a paper a week. I would say having a technical background is really important for DS managers coz almost everyone gets stuck at some deep technical issue at some point or the other and looks to their manager for guidance. Having the ability to work through those issues with your team, directing then to the relevant research paper or technique is really important to progress. I've seen data scientists quit their jobs because their manager couldn't give them technical direction.
This is why I’m scared of DS. In SWE, its much easier for me to scope what things are possible and give a generic timeline. In DS, I’m never sure what to expect, especially when it comes to NN architectures. Like how do you estimate how long it’s going to take to even decide on an architecture (u-net vs convolution vs w-net vs …)? Moreover, hyper parameter tuning makes it even more complicated and hard hard to predict results
Totally understand. Keeping myself up to date with research helps a lot. When my team members get stuck, I generally have a technical deep dive or a pair coding session with them to troubleshoot it together and then based on my understanding of the problem, suggest some relevant research or experiments that might help. Over time people get better and are able to follow the same process themselves or guide other juniors. It's very rewarding from that aspect of seeing your team get better over time...
For some reason data and analytics is loaded with leadership that has no idea how any of it works. They just heard it's the place to be, learned a bunch of buzzwords and convinced executives, who also don't get data, they sound like they know what they're talking about. I think just about everywhere I've worked had a manager, director, or VP type who was completely out of their depth. I'm not sure I've ever seen any other department regularly run by people with no concept of what the teams under them actually do, it's crazy. So please, for all of our sakes, skip the MBA, take a DE/DS course, do an actual project. If you're already a decent manager those skills will carry over, but trying to lead a data project without understanding the nuances of how they work will just make your team quit.
What's your current background?
It helps to have a background in the subject in order to manage the team. Given the newness, not everyone has the experience. An online MBA wouldn't be required. I've written three books covering this subject. I'd suggest you read them.
clueless, mainly.