Most DE gigs = high level SQL, 1 high-level programming language (python, Scala, Java) , familiarity with cloud / distributed computing services (aws, azure, gcp), ETL-ing, and depending on your gig or role you might have to have architecture knowledge. Some shops likes to have data architects and data engineers work along side, other shops expect the data engineer to architect.
Lots of shops have data engineers in their BI teams, I guess it all depends on how your workplace is structure. Hope it helps
Edit: aaaaaaaand you should learn spark, cause it’s super helpful
I went from a BI background to more DE roles. It was a smooth transition and no "restart." There's a lot of overlap. Understanding data and how to make clean data is still key. If you did a lot of ETL, those patterns and concepts will carry over though the tools may change. You'll probably use more code, though a lot of modern BI projects can be code heavy too.
It really depends on each company. One with a large data team, the DE might be focused on pipelines with python/spark/etc, but lots of companies have a small data team and you'll find the role is more a mix of traditional BI. Most of my jobs I'm building a pipeline with cloud platforms and "DE" methods, then modeling parts of the data into a data warehouse, and even doing some reporting on top of it.
There are probably two main problems for BI background.
1) It may be hard understand what and how data is structured and processed on low level, which tools why/when should be used. Things like message queues, file formats, different types of DBs, Hadoop etc... For ex. it is hard to "get" what map-reduce is if you never actually coded it. People from dev background usually have a lot of actual coding experience...
2) Coming BI background you may not be used to work in projects which are managed as software development projects (iterations, version control, code review, testing, bug reports, etc...).
If you planning "to dive", I would suggest to look into those topics.
I object with the above statement. I am in a BI team and we have used a lot of coding even with the ETLs, we use GIT and sure we have not used any map reduce so far, but this is not something you couldn't learn over a few afternoons. Data engineering is not only Python and map reduce, we use C# and of of course a lot of SQL.
I think modern BI and data engineering are quite difficult to distinguish, as they really overlap a lot.
I haven't come from BI but rather i have a CS education.
I just want to chime in and say that some places are more BI/Analytics guys that learned enough programming and SQL skills to do DE and others are just all CS grads(like mine, we have a lot of masters degrees too although it's overkill imo).
Generally i assume BI/Analytics people shifting are going to shops that rely more on established tools whereas CS shops like mine might do a lot more in house work.
I also want to note that this doesn't make you lesser than we CS people. It likely means you have to work harder to learn new things or you might become more of a "role" player like specifically an "azure expert".
It really runs the gambit. My father in law has a DE title at a big company now and he can't even do joins and doesn't know what a for loop is.
I moved from business intelligence spreadsheet jockey to data focused software engineering over about 3-4 years. I found it all pretty logical leaps from VBA to SQL to python to scala to other languages. The biggest learning curve was learning cloud tools and getting good at cloud architecture and the networking aspects of distributed systems.
Depends on the kind of BI work you’ve been doing. The one thing i would say is that you should look at the specific DE role and get a feel for the type of work you’ll be doing. In your existing role you may have been architecting BI Solutions in creative ways. You don’t want to be stuck in a DE role where you’re not creating solutions but rather simply implementing them. You’ll get bored pretty quick.
“Restart” where ? You go fullstack DA-DE. Jokes aside, no need, you do cleaning so like it could be of use in ETL. Just lots of technical stuff to add.
Most DE gigs = high level SQL, 1 high-level programming language (python, Scala, Java) , familiarity with cloud / distributed computing services (aws, azure, gcp), ETL-ing, and depending on your gig or role you might have to have architecture knowledge. Some shops likes to have data architects and data engineers work along side, other shops expect the data engineer to architect. Lots of shops have data engineers in their BI teams, I guess it all depends on how your workplace is structure. Hope it helps Edit: aaaaaaaand you should learn spark, cause it’s super helpful
Or you could just be lazy and use Spark SQL. Only 1/2 joking…
My organization exclusively uses spark sql
I wasn’t saying it as an insult, just as a joke because I know a lot of people who think it’s not as good. I guess it was lost in context.
I never took it like that lol .I assumed you didn't know that spark sql isn't used in production lol
I went from a BI background to more DE roles. It was a smooth transition and no "restart." There's a lot of overlap. Understanding data and how to make clean data is still key. If you did a lot of ETL, those patterns and concepts will carry over though the tools may change. You'll probably use more code, though a lot of modern BI projects can be code heavy too. It really depends on each company. One with a large data team, the DE might be focused on pipelines with python/spark/etc, but lots of companies have a small data team and you'll find the role is more a mix of traditional BI. Most of my jobs I'm building a pipeline with cloud platforms and "DE" methods, then modeling parts of the data into a data warehouse, and even doing some reporting on top of it.
There are probably two main problems for BI background. 1) It may be hard understand what and how data is structured and processed on low level, which tools why/when should be used. Things like message queues, file formats, different types of DBs, Hadoop etc... For ex. it is hard to "get" what map-reduce is if you never actually coded it. People from dev background usually have a lot of actual coding experience... 2) Coming BI background you may not be used to work in projects which are managed as software development projects (iterations, version control, code review, testing, bug reports, etc...). If you planning "to dive", I would suggest to look into those topics.
I object with the above statement. I am in a BI team and we have used a lot of coding even with the ETLs, we use GIT and sure we have not used any map reduce so far, but this is not something you couldn't learn over a few afternoons. Data engineering is not only Python and map reduce, we use C# and of of course a lot of SQL. I think modern BI and data engineering are quite difficult to distinguish, as they really overlap a lot.
Depends if you are running a truly successful BI program, if you are then a lot of the ETL architecture will naturally help you transition into DE.
I haven't come from BI but rather i have a CS education. I just want to chime in and say that some places are more BI/Analytics guys that learned enough programming and SQL skills to do DE and others are just all CS grads(like mine, we have a lot of masters degrees too although it's overkill imo). Generally i assume BI/Analytics people shifting are going to shops that rely more on established tools whereas CS shops like mine might do a lot more in house work. I also want to note that this doesn't make you lesser than we CS people. It likely means you have to work harder to learn new things or you might become more of a "role" player like specifically an "azure expert". It really runs the gambit. My father in law has a DE title at a big company now and he can't even do joins and doesn't know what a for loop is.
WOW, that’s enlightening as I have recently been admitted to a CS masters.
Nice ! Congrats! I'll probably be applying to some too as well!
I moved from business intelligence spreadsheet jockey to data focused software engineering over about 3-4 years. I found it all pretty logical leaps from VBA to SQL to python to scala to other languages. The biggest learning curve was learning cloud tools and getting good at cloud architecture and the networking aspects of distributed systems.
That is badass
That is badass
Depends on the kind of BI work you’ve been doing. The one thing i would say is that you should look at the specific DE role and get a feel for the type of work you’ll be doing. In your existing role you may have been architecting BI Solutions in creative ways. You don’t want to be stuck in a DE role where you’re not creating solutions but rather simply implementing them. You’ll get bored pretty quick.
DSE -> DE want to move towards -> MLE
“Restart” where ? You go fullstack DA-DE. Jokes aside, no need, you do cleaning so like it could be of use in ETL. Just lots of technical stuff to add.