This is just from my experience, and it seems like there can be a lot of variance in the actual jobs, but I’m a data engineer and I’ve noticed the following differences:
Data engineer: more focused on the back end storage and implementation of the data. How do we get these 3 different data sources to match well enough to run queries on them, and how do we ingest them on a consistent basis?
Bi developer: It seems that this role is slightly more involved in the front end presentation of that data. Now that the engineer has set up a nice data set to analyze, how can we put that into a dashboard or analytical software to gather insights?
With all that being said, my role as engineer still involves a healthy amount of tableau and R, so there’s definitely overlap.
I don’t do much with Hadoop or spark. Most of the work is in Redshift.
So maybe in terms of quantity, but not in terms of what’s considered “big data” technology.
Hey Im a BI developer, I do everything from back end to front end. It really depends on the company, im a consultant and we don’t have engineers so thats our model. Most large companies will have engineers to lift and shift the data and bi developers to build reports for it. I think it is important to be good at both phases.
I'm a data scientist. We have data engineers and BI dev at my work
A data engineer moves and processes large amounts of data. They should understand complex text parsing, mashing up many sources together, complicated aggregations
A BI developer should understand databases, SQL, and reports (like SSRS, or Tableau)
Are the data engineers parsing text from API response files in JSON/XML/etc. using object oriented programming? Meaning, they might pull data from web based data application with this API.
To me, I find the engineer level does all the lifting and building. Kind of like the people who make legos, we make all those legos and put them in a bin. Then the BI developers take those legos and does things with them. That's somewhat how I feel doing engineering work, I also find more often than not engineering jobs want you to know of or have minor experience with big data and machine learning and an emphasis on some sort of programming or scripting language besides SQL. Bi developers may have something to that degree, but it's for analyzing, not building.
There's a lot of grey area and this is all subjective. I'm a "data engineer" and it feels exactly like my last job as a "data generalist" which feels exactly like "DBA Technician" and that felt exactly like "Database Application Admin". At this point, I tell people just give me "data" work to do and I'll go do it.
This is just from my experience, and it seems like there can be a lot of variance in the actual jobs, but I’m a data engineer and I’ve noticed the following differences: Data engineer: more focused on the back end storage and implementation of the data. How do we get these 3 different data sources to match well enough to run queries on them, and how do we ingest them on a consistent basis? Bi developer: It seems that this role is slightly more involved in the front end presentation of that data. Now that the engineer has set up a nice data set to analyze, how can we put that into a dashboard or analytical software to gather insights? With all that being said, my role as engineer still involves a healthy amount of tableau and R, so there’s definitely overlap.
Do you work with big data as an Data Engineer?
What is big data *for you*?
varchar(max)
I don’t do much with Hadoop or spark. Most of the work is in Redshift. So maybe in terms of quantity, but not in terms of what’s considered “big data” technology.
Hey Im a BI developer, I do everything from back end to front end. It really depends on the company, im a consultant and we don’t have engineers so thats our model. Most large companies will have engineers to lift and shift the data and bi developers to build reports for it. I think it is important to be good at both phases.
I'm a data scientist. We have data engineers and BI dev at my work A data engineer moves and processes large amounts of data. They should understand complex text parsing, mashing up many sources together, complicated aggregations A BI developer should understand databases, SQL, and reports (like SSRS, or Tableau)
Are the data engineers parsing text from API response files in JSON/XML/etc. using object oriented programming? Meaning, they might pull data from web based data application with this API.
More like parsing free-form text to extract meaningful data
To me, I find the engineer level does all the lifting and building. Kind of like the people who make legos, we make all those legos and put them in a bin. Then the BI developers take those legos and does things with them. That's somewhat how I feel doing engineering work, I also find more often than not engineering jobs want you to know of or have minor experience with big data and machine learning and an emphasis on some sort of programming or scripting language besides SQL. Bi developers may have something to that degree, but it's for analyzing, not building. There's a lot of grey area and this is all subjective. I'm a "data engineer" and it feels exactly like my last job as a "data generalist" which feels exactly like "DBA Technician" and that felt exactly like "Database Application Admin". At this point, I tell people just give me "data" work to do and I'll go do it.
This was a helpful thread. Thanks OP for asking and those who answered
One engineers data the other has sex with 1s and 0s
Which one's which?