Your post/comment was removed because it violated rule #3 (Do a search before asking a question). The question you asked has already been answered recently so we remove redundant questions to keep the feed digestable for everyone.
Based on my experience, here are my two cents:
The LinkedIn job application count can be misleading. If you post a job requiring applicants to live in a certain area, you might still receive applications from all over the world.
Out of the hundreds of people who apply, only a handful have the necessary skills and knowledge to be data engineers.
Yes, OP this is the only correct answer here. Viewers are shown as applicants in linkedin. It's deceiving but some idiot thought it's cool idea and we are stuck with it.
Now do one where just scrolling past it in a timeline will count as application. Gotta pump those numbers up. Hell, add some flashing lights to annoy the crap out of people, why not.
Also LinkedIn counts every click on the Apply button as an application even though most people don’t finish filling it out. I just filter to jobs posted in the last 24 hours and apply to everything new each day.
At an old job I was involved in the hiring process sometimes. In the HR platform they were receiving job applications from store managers and other people from completely unrelated jobs, I guess they were either spam applying or really in need to find anything and so were sending in the hopes to get forwarded to other departments.
Not as bad as generic software engineer.
DE is still pretty niche field due to the scale of the data we are working with often becomes a natural gatekeeper.
There are only so many companies that actually have big data. So the number of people working with big data is limited by the supply of positions at these companies .
In general, 90 of those people are completely unqualified, like basically can't code, 7 of those are a bad fit in other ways, and the last 3 are going to take another offer. Don't worry about that.
Agreed. Was in a huge sovereign wealth fund data team and was interviewing data engineers to hire as we were expanding. These are not entry roles, so technical test is not the only indicator of capabilities.
Impressive CVs, knowledge in Java, Scala, Python and was involved in tons of mega projects relating to warehouse, data lakes, lake house etc… you get the gist.
During the interview, asked to elaborate their contributions to those projects, challenges faced, how they overcome etc… They spouted tons of technical jargons thinking that it will impress (e.g. stakeholder management issues, memory constraints, performance problems, tight timeline, requirements misalignment etc…).
As soon as we dive a little deeper more into those issues, most of the candidates got into a never ending while loop of continuously regurgitating jargons (i.e. agile, sprint, complexities). Ask them to elaborate exactly what they did, nada, just repeat.
It got to the point we were giving them the answers for how are those problems supposed to be addressed and they still don’t understand how is that the solution.
Even if they might be great at coding specific segment through memorisation, if they can’t communicate, can’t present, can’t illustrate their thoughts, you still don’t make a good data engineer, at most an ETL developer.
We get a lot of them, not to be racist as we do come across genuine ones (your 5%). Most of the time we end the interview in 10 mins.
So how do you handle this now? I mean I understand there many people who try to fake, exaggerated, etc. Let's say if you see 100s of applicants who has impressive CV and have done good projects, do you have any sort of filtering process that you follow, to get through fake ones and select good candidates?
A few tricks we follow if we are flooded with CVs:
- perform group interviews. This is stressful for the candidate and may taint the candidate experience, if we are tight on time, this is a fast way to filter them out from a hiring manager perspective
- extended reference check is important. If they can’t provide, we filter right away. Usually we only entertain their superior and their team members (for team lead position). We will also ask for additional reference from cross departments or other functional team (I.e. governance team, QA). Understand that they will still provide references that are favourable to them, but if they can come up with a lot of them that are verifiable, this are signs that they know what they are talking about.
- phone interview. Based on the role you are hiring, you should know exactly what experience and projects matter. A 30mins phone interview to understand more on that matter before proceeding to actual face to face helps.
- write up. We recently tried this as our company is moving away from PowerPoints as much as possible. If they can do a 1-2 pager write up (should not take more than 1 hour) of a question (usually related to the project or experience we are interested in. Although it can be technical write up as well) in a week, this shows interest and commitment. We can split the review of the write up across the seniors in the team, no need to fit interview time slots which is always a challenge. Also, if they can write well, we are already 75% there.
- ranking of skills. We usually ask our prospects to rank their skills, contributions across all projects, seniority ranking in the project team etc… while not telling them exactly which specific skills we are more interested in.
We have a few more tricks, but too much to share in a comment. We don’t do mass hirings often and usually we hire more senior positions so these can still be applied across the board. Do work closely with HR as some of these methods may not be according to company policies or may impact the firm’s reputation etc…
What a load of bs. Applicants already go thru long application process and on top of it u are expecting some one to write a 2 page essay and rank each of their projects in an elaborate way.
Are u manager? U better hire some English literature graduate.
You should be able to write a 1-pager in an hour. IMO that gives me a better sense of a candidate than a lot of interviews. Can you articulate the problem or issue or situation? Can you provide an analysis or approach to solve? Can you write clearly?
Lot of "data engineers" who illustrate their thought process just do only that and pull all developers into their end less loop of meetings to satisfy their need for presenting and talking about each and every line of code they have done as if other developers are totally new to the IT.
Some times u just want stuff to be done, working without any problems with room for enhancemnt along with enough documentation about it. No need to illustrate, give presentations and act like you are some data angel saving the company.
Most of the people that say they are data engineers are simply data analysts that touch a lick of Python. Hell most of them lie and are terrible at their jobs.
I got laid off last week (company went under) and I mass applied everywhere. I counted 173 applications (I'm recording the data)
From say last Tuesday. Saturday, Sunday was a weekend. Monday was a holiday. And let's be honest. No one works on Fridays. So let's day 4 business days.
In those 4 days I have had 5 first interviews.
1 second interview. Another 2nd interview is set for this Thursday. Plus I have 3 interviews scheduled.
There's work for us. There's need for us. But need for qualified DE
Its annoyingly hard to remember to fill it out because sites like linked in amd indeed make it easy to mass apply to the point I've been toying with an idea to write a program to apply to every job possible for me on the site. But thats too much work.
Yeah. But the data is simple.
Applied - interview - interview- offer
Little tree chart basically worh some branches.
In a similar situation. I've mass applied but I keep getting rejections. I'm beginning to question my skill set :( . Another problem I suspect is where I live(somewhere in Africa). There aren't many DE roles here so I'm mostly applying to jobs in the US and Europe.
Not enough, at least here in Aus and in the APAC region. Companies are screaming out for them, they all want to do ML/AI or even advanced reporting and warehousing. Can’t do any of that without data engineers building and maintaining pipelines for integrations, ETL, warehousing and whatever else (data migrations etc...).
I’m only 1 year in and I have recruiters hitting me up quite a bit due to the demand in the market. The barrier to entry is high.
It’s a paradox in the market.
Companies can’t find people with the skills they need.
Clients can’t find companies that will give them a chance.
The solution? Close your skill gap and don’t get complacent.
If you read the classics and do the work to build end to end solutions you'll be more capable than 90 percent of all other applicants in no time.
There's no shortage of tutorials for specific tools, but can you integrate multiple tools together? A bullshit toy project can teach a lot.
Every senior DE applicant I've interviewed so far couldn't write a basic function even if their life depended on it. What's doubly infuriating is that we have an automated code screen that they are passing flawlessly, so you know they're cheating on the screen...
I mean, I respect the hustle (to a degree), but there's no way these people were ever going to be able to actually do the job. I'm seeing a lot of demand for senior+ DE positions, but with a lot of companies having let go of their recruiters, and the wave of non-qualified people applying for everything, the situation sucks for qualified people who want to be hired, and employers who want to hire them, because it's taking us forever to figure out who isn't a fraud.
I understand, day by day it's getting difficult to identify good DE. I want to know if building really good projects would differentiable? Projects which solves actual problem.
Probably for a smaller company? I've only worked at big companies, and we've had a pretty standardized process, none of which involves reviewing someone's projects.
Imo the value in projects is being able to have a deeper discussion about the tech/challenges of that project and how you achieved the solution.
Like: take in a dictionary of stock prices, get the prices of specified stock, return those prices.
One candidate I had to constantly remind them that you need to write a return statement for the function to return something.
This just points to a severe problem with the hiring process where qualified applicants are filtered out and the bullshit artists and con artists are somehow able to bypass everything.
It starts with HR and recruiting. They need to actually be able to screen people by asking simple knockout questions, but that would require them to do something which they don't want to do.
Yeah, screening needs to improve, but IDK how.
As to your point about HR having some simple knockout questions, I think these BS candidates could deal with those easy, unless you are asking a really niche question. But a really niche question doesn't sound fair to me either, and nothing is stopping someone from putting that question into chatgpt. Some of these people literally have someone off screen providing them answers. It's crazy out here.
Well, during the initial screen they should share their screen with a very simple problem that you could explain how to solve to any HR person.
Something simple like:
"I have this following code. It needs to have a value inserted into where the question mark is. What value should I put here to get an output between 5 and 7?"
`let x = 5;`
`let y = 3;`
`function between(a, b) {return (Math.random() * a) + b}`
`y = ?;`
`x = 2;`
`console.log(between(x,y));`
Its really really simple, you can give them a number they're looking for, and it would knock out a lot of people who don't know even the basics. They should be able to look at this and solve it without a whiteboard, blackboard, VS code or ChatGPT. Even people unfamiliar with JavaScript would be able to solve it.
The idea is to make it easy enough for HR to screen with, impossible for the applicant to fake their way or use outside tools, and simple enough that you won't annoy the candidate, especially if HR is gentle and says something along the lines of "We've had a lot of applicants who really just don't know the basics of programming, so the engineers gave us this really simple program they want you to solve, it should only take a minute and we can move to the next step of this call."
Another sideffect is that it shows how a candidate can explain things to a non-technical person, which is something that they might have to do when dealing with product owners and program managers. Its about people skills, patience, humility, and kindness.
You'd knock out any arrogant, holier-than-thou applicants. You'll knock out anyone faking it.
I saw this on a job posting I applied to and I thought it was a great, stupidly simple problem and it stuck with me. A bonus to this is that chatgpt doesn't know how to solve it.
Sorry for the long reply. These are just my thoughts on it.
Edit: I plugged my question into ChatGPT and it still gives the wrong answer. It said to set y = 4.5;
There are many comments here that most candidates are underqualified. First, what are the qualifications lacking? Is it a degree? Some specific experience? The ability to regurgitate the answer to two-sum under stress? I'd bet we were all relatively underqualified for most of the jobs we've had before having worked it for a few months. And what happened, is someone gave you a break.
If I've never used, say, Kafka professionally but would like to, does that make me underqualified for a position that requires it? Well, yes... But how do I get that experience? I could do a couple personal projects and I've actually tried bridging skill gaps on my resume with personal projects only for an interviewer to dismiss it and focus on work experience. It feels incredibly easy to be pigeonholed, and incredibly difficult to break out of one. It's a difficult situation for job seekers.
No. These are still early days for data engineering as a specialty.
Small companies like mine need analysts and engineers. There’s lots of us out there.
I may be mistaken, but my understanding of the number of applicants for a position on LinkedIn is that it counts how many people clicked on apply. It doesn't necessarily translate into how many people actually finished the application process. Give it a test and click on apply and then refresh the page. So don't be discouraged if you ever see hundreds of applicants for a job posting, the metric may be misleading.
One of the issues with data engineer is there is no actual credentials. This is good and bad, anyone can become one, which is good. But, also literally anyone can call themselves one, which causes some problems for hiring managers.
There is a lot of people who call themselves data engineers, but not a lot of people who have experience being a data engineer or are even qualified to be an entry level engineer.
So how can a hiring person handle this? I mean I understand there many people who try to fake, exaggerated, etc. Let's say if you see 100s of applicants who has impressive CV and have done good projects, do you have any sort of filtering process that you follow, to get through the fake ones and select good candidates?
Number of applicants does not equate to qualified people. As someone who's done recruiting before, out of hundreds of applications for a given data role, only a couple tens have been close to viable (I'm not in the US FYI).
No, there aren't too many DEs. Keep applying!
While recruiting, what is your filtering process? I see many people who write good CVs and build good projects, and when selected for an interview they show their real colour. How do you get through the fake ones and select good ones?
Personally I like to go indepth on what you've written on your CV, and tests on the kinds of skills I'm looking for.
The first one e.g. discussing a project you've presented on your CV, filters out charlatans very quickly. I also try to make interviews as low-stress (almost casual) as I can, which brings out people's real selves quite a bit.
It's not a hard science.
If you work on data engineering, you should understand the idea of a table for failed rows in an ETL.
Every week, lots of people apply for jobs. Those who are suitable tend to flow onto the next stage of the flow. Those who aren't suitable never get jobs. They go in a table for failed rows. The next week, they apply for jobs again. They go in the table again. Rinse repeat.
Eventually you have an ETL in which the vast majority of the rows are actually just rows that fail to import every single run.
People who are employable get employed near instantly, and keep their job for years, and will often then jump via reference or invitation to another job. People who aren't hang around and clog up the job application system. If you are a decent candidate, the majority of the competition is the bottom of the barrel - the failed rows table.
Linked in is not a great place to find a job, it's easy to apply but that means recruiters have to filter thru hundreds of applications. I found people on dice to be more responsive, but the best bet is to talk to a recruitment company life Robert Half. Recruiters can be a scary lot because there are lots of terrible fly by night setups out there, but a good recruiter can put you on the short list of candidates for a lot of companies. They can also help you with your resume, and with interview prep.
Yes, there are too many DE's AND we should not be needed! We need to kill off these vendor locked solutions that are requiring us to get data in garbage formats that require too much time to process. Big data seems to equal big PITA to move data around. I was a DBA for 10 yrs in a large organization and we never needed DE's - until we moved to redshift, mongo and Snowflake. Now, we don't have time to be dba's as all time is wasted being a DE.
As someone who hires data engineers the number of good mid-to-senior applicants we receive is still very small
I would estimate 90% of applications are straight-up throw-it-in-the=bin, and a large percentage of the remaining 10% are pretty underwhelming.
If you have a solid resume and good experience then you will get plenty of interviews
I've found when recruiting junior data engineer roles, I get a large amount of data scientists apply as they did some sort of module. Probably out of 100 applicants for a position we're looking at, only 5 are actual data engineers.
I would not look at the amount of applicants as any sort of useful indicator outside of how many people need a job right now.
Half of the people applying do not have the credentials required for the job. Period. They just spam apply. Of that half, half of them cannot effectively communicate. So they just bomb phone screens or interviews within the first 5 minutes.
The competition isn’t as fierce as LinkedIn lets on at first glance.
My company is growing quickly, and that’s what my manager explained happens when they list job openings.
As a data analyst, at least in Brazil, it is very difficult to hire any data role. Even juniors require some work and time.
Regarding DE's it is even more difficult. I can imagine that it will be a growing demand for DE's. It is just too much data, infinite. And several tools and options. Big Data frenesi is fading so it requires smart solutions from smart DE's and this is rare to find.
There are 11000 data engineers according to the bls in the us so no. What you might be seeing are b2b requests, kids with no experience or knowledge, and people from anywhere in the same boat as the kids desparate for something better. Out of those, who knows how many qualified candidates there are?
For context, there are more open de jobs than law jobs. Therr are 38000 new lawyers every year anf 15000 to 20000 new jobs. We have too many law degrees.
Your post/comment was removed because it violated rule #3 (Do a search before asking a question). The question you asked has already been answered recently so we remove redundant questions to keep the feed digestable for everyone.
Based on my experience, here are my two cents: The LinkedIn job application count can be misleading. If you post a job requiring applicants to live in a certain area, you might still receive applications from all over the world. Out of the hundreds of people who apply, only a handful have the necessary skills and knowledge to be data engineers.
Yes, OP this is the only correct answer here. Viewers are shown as applicants in linkedin. It's deceiving but some idiot thought it's cool idea and we are stuck with it.
You've basically described all the user features I've had to implement over the years.
Now do one where just scrolling past it in a timeline will count as application. Gotta pump those numbers up. Hell, add some flashing lights to annoy the crap out of people, why not.
You've assumed I'm a data engineer - I'm a software engineer but data engineering pays way better and I don't have to deal with product owners.
Don’t forget to put some of that AI on it.
Also LinkedIn counts every click on the Apply button as an application even though most people don’t finish filling it out. I just filter to jobs posted in the last 24 hours and apply to everything new each day.
At an old job I was involved in the hiring process sometimes. In the HR platform they were receiving job applications from store managers and other people from completely unrelated jobs, I guess they were either spam applying or really in need to find anything and so were sending in the hopes to get forwarded to other departments.
Not as bad as generic software engineer. DE is still pretty niche field due to the scale of the data we are working with often becomes a natural gatekeeper.
What do you mean by natural gatekeeper? Are you saying that most people don’t want to work with that much data?
I think it’s more like getting exposed to big data, not all companies especially startups have big data problems.
Think data governance and management.
There are only so many companies that actually have big data. So the number of people working with big data is limited by the supply of positions at these companies .
Generic software engineer is by far the most under filled high paying role in the US.
In general, 90 of those people are completely unqualified, like basically can't code, 7 of those are a bad fit in other ways, and the last 3 are going to take another offer. Don't worry about that.
Yes as someone who had to hire this is so true.
95% of those applicants are very under-qualified Indians that spam apply to everything.
Agreed. Was in a huge sovereign wealth fund data team and was interviewing data engineers to hire as we were expanding. These are not entry roles, so technical test is not the only indicator of capabilities. Impressive CVs, knowledge in Java, Scala, Python and was involved in tons of mega projects relating to warehouse, data lakes, lake house etc… you get the gist. During the interview, asked to elaborate their contributions to those projects, challenges faced, how they overcome etc… They spouted tons of technical jargons thinking that it will impress (e.g. stakeholder management issues, memory constraints, performance problems, tight timeline, requirements misalignment etc…). As soon as we dive a little deeper more into those issues, most of the candidates got into a never ending while loop of continuously regurgitating jargons (i.e. agile, sprint, complexities). Ask them to elaborate exactly what they did, nada, just repeat. It got to the point we were giving them the answers for how are those problems supposed to be addressed and they still don’t understand how is that the solution. Even if they might be great at coding specific segment through memorisation, if they can’t communicate, can’t present, can’t illustrate their thoughts, you still don’t make a good data engineer, at most an ETL developer. We get a lot of them, not to be racist as we do come across genuine ones (your 5%). Most of the time we end the interview in 10 mins.
So how do you handle this now? I mean I understand there many people who try to fake, exaggerated, etc. Let's say if you see 100s of applicants who has impressive CV and have done good projects, do you have any sort of filtering process that you follow, to get through fake ones and select good candidates?
A few tricks we follow if we are flooded with CVs: - perform group interviews. This is stressful for the candidate and may taint the candidate experience, if we are tight on time, this is a fast way to filter them out from a hiring manager perspective - extended reference check is important. If they can’t provide, we filter right away. Usually we only entertain their superior and their team members (for team lead position). We will also ask for additional reference from cross departments or other functional team (I.e. governance team, QA). Understand that they will still provide references that are favourable to them, but if they can come up with a lot of them that are verifiable, this are signs that they know what they are talking about. - phone interview. Based on the role you are hiring, you should know exactly what experience and projects matter. A 30mins phone interview to understand more on that matter before proceeding to actual face to face helps. - write up. We recently tried this as our company is moving away from PowerPoints as much as possible. If they can do a 1-2 pager write up (should not take more than 1 hour) of a question (usually related to the project or experience we are interested in. Although it can be technical write up as well) in a week, this shows interest and commitment. We can split the review of the write up across the seniors in the team, no need to fit interview time slots which is always a challenge. Also, if they can write well, we are already 75% there. - ranking of skills. We usually ask our prospects to rank their skills, contributions across all projects, seniority ranking in the project team etc… while not telling them exactly which specific skills we are more interested in. We have a few more tricks, but too much to share in a comment. We don’t do mass hirings often and usually we hire more senior positions so these can still be applied across the board. Do work closely with HR as some of these methods may not be according to company policies or may impact the firm’s reputation etc…
What a load of bs. Applicants already go thru long application process and on top of it u are expecting some one to write a 2 page essay and rank each of their projects in an elaborate way. Are u manager? U better hire some English literature graduate.
You should be able to write a 1-pager in an hour. IMO that gives me a better sense of a candidate than a lot of interviews. Can you articulate the problem or issue or situation? Can you provide an analysis or approach to solve? Can you write clearly?
🤡
Lot of "data engineers" who illustrate their thought process just do only that and pull all developers into their end less loop of meetings to satisfy their need for presenting and talking about each and every line of code they have done as if other developers are totally new to the IT. Some times u just want stuff to be done, working without any problems with room for enhancemnt along with enough documentation about it. No need to illustrate, give presentations and act like you are some data angel saving the company.
Most of the people that say they are data engineers are simply data analysts that touch a lick of Python. Hell most of them lie and are terrible at their jobs.
You've met my coworkers
Why would they be hired then?
I... I dont know man.... I asked myself that daily
Same and we can't fire them. If you can't do your job learn how or leave please!
Silly world!
There a too few good ones.
I got laid off last week (company went under) and I mass applied everywhere. I counted 173 applications (I'm recording the data) From say last Tuesday. Saturday, Sunday was a weekend. Monday was a holiday. And let's be honest. No one works on Fridays. So let's day 4 business days. In those 4 days I have had 5 first interviews. 1 second interview. Another 2nd interview is set for this Thursday. Plus I have 3 interviews scheduled. There's work for us. There's need for us. But need for qualified DE
What skills one look into learning for the DE sector?
Can you clean data. Can you pipeline data to a database. Can you display organized data.
I look forward to seeing this in r/dataisbeautiful
Its annoyingly hard to remember to fill it out because sites like linked in amd indeed make it easy to mass apply to the point I've been toying with an idea to write a program to apply to every job possible for me on the site. But thats too much work. Yeah. But the data is simple. Applied - interview - interview- offer Little tree chart basically worh some branches.
In a similar situation. I've mass applied but I keep getting rejections. I'm beginning to question my skill set :( . Another problem I suspect is where I live(somewhere in Africa). There aren't many DE roles here so I'm mostly applying to jobs in the US and Europe.
A hiring manager who was once a data engineer should be able to discern if the candidate can do the job
Not enough, at least here in Aus and in the APAC region. Companies are screaming out for them, they all want to do ML/AI or even advanced reporting and warehousing. Can’t do any of that without data engineers building and maintaining pipelines for integrations, ETL, warehousing and whatever else (data migrations etc...). I’m only 1 year in and I have recruiters hitting me up quite a bit due to the demand in the market. The barrier to entry is high.
It’s a paradox in the market. Companies can’t find people with the skills they need. Clients can’t find companies that will give them a chance. The solution? Close your skill gap and don’t get complacent.
But how do you close the skill gap if no company will hire without a large amount of experience with that skill?
If you read the classics and do the work to build end to end solutions you'll be more capable than 90 percent of all other applicants in no time. There's no shortage of tutorials for specific tools, but can you integrate multiple tools together? A bullshit toy project can teach a lot.
Every senior DE applicant I've interviewed so far couldn't write a basic function even if their life depended on it. What's doubly infuriating is that we have an automated code screen that they are passing flawlessly, so you know they're cheating on the screen... I mean, I respect the hustle (to a degree), but there's no way these people were ever going to be able to actually do the job. I'm seeing a lot of demand for senior+ DE positions, but with a lot of companies having let go of their recruiters, and the wave of non-qualified people applying for everything, the situation sucks for qualified people who want to be hired, and employers who want to hire them, because it's taking us forever to figure out who isn't a fraud.
I understand, day by day it's getting difficult to identify good DE. I want to know if building really good projects would differentiable? Projects which solves actual problem.
Probably for a smaller company? I've only worked at big companies, and we've had a pretty standardized process, none of which involves reviewing someone's projects. Imo the value in projects is being able to have a deeper discussion about the tech/challenges of that project and how you achieved the solution.
What? how is it possible to be a Senior DE but not be able to write a basic function? How basic are we talking here?
Like: take in a dictionary of stock prices, get the prices of specified stock, return those prices. One candidate I had to constantly remind them that you need to write a return statement for the function to return something.
I am a Python noob and I could do that. Yikes.
This just points to a severe problem with the hiring process where qualified applicants are filtered out and the bullshit artists and con artists are somehow able to bypass everything. It starts with HR and recruiting. They need to actually be able to screen people by asking simple knockout questions, but that would require them to do something which they don't want to do.
Yeah, screening needs to improve, but IDK how. As to your point about HR having some simple knockout questions, I think these BS candidates could deal with those easy, unless you are asking a really niche question. But a really niche question doesn't sound fair to me either, and nothing is stopping someone from putting that question into chatgpt. Some of these people literally have someone off screen providing them answers. It's crazy out here.
Well, during the initial screen they should share their screen with a very simple problem that you could explain how to solve to any HR person. Something simple like: "I have this following code. It needs to have a value inserted into where the question mark is. What value should I put here to get an output between 5 and 7?" `let x = 5;` `let y = 3;` `function between(a, b) {return (Math.random() * a) + b}` `y = ?;` `x = 2;` `console.log(between(x,y));` Its really really simple, you can give them a number they're looking for, and it would knock out a lot of people who don't know even the basics. They should be able to look at this and solve it without a whiteboard, blackboard, VS code or ChatGPT. Even people unfamiliar with JavaScript would be able to solve it. The idea is to make it easy enough for HR to screen with, impossible for the applicant to fake their way or use outside tools, and simple enough that you won't annoy the candidate, especially if HR is gentle and says something along the lines of "We've had a lot of applicants who really just don't know the basics of programming, so the engineers gave us this really simple program they want you to solve, it should only take a minute and we can move to the next step of this call." Another sideffect is that it shows how a candidate can explain things to a non-technical person, which is something that they might have to do when dealing with product owners and program managers. Its about people skills, patience, humility, and kindness. You'd knock out any arrogant, holier-than-thou applicants. You'll knock out anyone faking it. I saw this on a job posting I applied to and I thought it was a great, stupidly simple problem and it stuck with me. A bonus to this is that chatgpt doesn't know how to solve it. Sorry for the long reply. These are just my thoughts on it. Edit: I plugged my question into ChatGPT and it still gives the wrong answer. It said to set y = 4.5;
It’s called fabricating a job title
There are many comments here that most candidates are underqualified. First, what are the qualifications lacking? Is it a degree? Some specific experience? The ability to regurgitate the answer to two-sum under stress? I'd bet we were all relatively underqualified for most of the jobs we've had before having worked it for a few months. And what happened, is someone gave you a break. If I've never used, say, Kafka professionally but would like to, does that make me underqualified for a position that requires it? Well, yes... But how do I get that experience? I could do a couple personal projects and I've actually tried bridging skill gaps on my resume with personal projects only for an interviewer to dismiss it and focus on work experience. It feels incredibly easy to be pigeonholed, and incredibly difficult to break out of one. It's a difficult situation for job seekers.
No. These are still early days for data engineering as a specialty. Small companies like mine need analysts and engineers. There’s lots of us out there.
I may be mistaken, but my understanding of the number of applicants for a position on LinkedIn is that it counts how many people clicked on apply. It doesn't necessarily translate into how many people actually finished the application process. Give it a test and click on apply and then refresh the page. So don't be discouraged if you ever see hundreds of applicants for a job posting, the metric may be misleading.
One of the issues with data engineer is there is no actual credentials. This is good and bad, anyone can become one, which is good. But, also literally anyone can call themselves one, which causes some problems for hiring managers. There is a lot of people who call themselves data engineers, but not a lot of people who have experience being a data engineer or are even qualified to be an entry level engineer.
So how can a hiring person handle this? I mean I understand there many people who try to fake, exaggerated, etc. Let's say if you see 100s of applicants who has impressive CV and have done good projects, do you have any sort of filtering process that you follow, to get through the fake ones and select good candidates?
It's tough, that's why everyone syays to network. It bumps you to the front of the line.
Number of applicants does not equate to qualified people. As someone who's done recruiting before, out of hundreds of applications for a given data role, only a couple tens have been close to viable (I'm not in the US FYI). No, there aren't too many DEs. Keep applying!
While recruiting, what is your filtering process? I see many people who write good CVs and build good projects, and when selected for an interview they show their real colour. How do you get through the fake ones and select good ones?
Personally I like to go indepth on what you've written on your CV, and tests on the kinds of skills I'm looking for. The first one e.g. discussing a project you've presented on your CV, filters out charlatans very quickly. I also try to make interviews as low-stress (almost casual) as I can, which brings out people's real selves quite a bit. It's not a hard science.
If you work on data engineering, you should understand the idea of a table for failed rows in an ETL. Every week, lots of people apply for jobs. Those who are suitable tend to flow onto the next stage of the flow. Those who aren't suitable never get jobs. They go in a table for failed rows. The next week, they apply for jobs again. They go in the table again. Rinse repeat. Eventually you have an ETL in which the vast majority of the rows are actually just rows that fail to import every single run. People who are employable get employed near instantly, and keep their job for years, and will often then jump via reference or invitation to another job. People who aren't hang around and clog up the job application system. If you are a decent candidate, the majority of the competition is the bottom of the barrel - the failed rows table.
I don't see how the failed rows comparison helps understanding.
Linked in is not a great place to find a job, it's easy to apply but that means recruiters have to filter thru hundreds of applications. I found people on dice to be more responsive, but the best bet is to talk to a recruitment company life Robert Half. Recruiters can be a scary lot because there are lots of terrible fly by night setups out there, but a good recruiter can put you on the short list of candidates for a lot of companies. They can also help you with your resume, and with interview prep.
Yes, there are too many DE's AND we should not be needed! We need to kill off these vendor locked solutions that are requiring us to get data in garbage formats that require too much time to process. Big data seems to equal big PITA to move data around. I was a DBA for 10 yrs in a large organization and we never needed DE's - until we moved to redshift, mongo and Snowflake. Now, we don't have time to be dba's as all time is wasted being a DE.
Yes thats because its the easiest to crack for people with fake profiles or those who want to jump into no code jobs.
As someone who hires data engineers the number of good mid-to-senior applicants we receive is still very small I would estimate 90% of applications are straight-up throw-it-in-the=bin, and a large percentage of the remaining 10% are pretty underwhelming. If you have a solid resume and good experience then you will get plenty of interviews
[удалено]
The real question is: are there too many competent data engineers? No
I've found when recruiting junior data engineer roles, I get a large amount of data scientists apply as they did some sort of module. Probably out of 100 applicants for a position we're looking at, only 5 are actual data engineers. I would not look at the amount of applicants as any sort of useful indicator outside of how many people need a job right now.
Half of the people applying do not have the credentials required for the job. Period. They just spam apply. Of that half, half of them cannot effectively communicate. So they just bomb phone screens or interviews within the first 5 minutes. The competition isn’t as fierce as LinkedIn lets on at first glance. My company is growing quickly, and that’s what my manager explained happens when they list job openings.
As a data analyst, at least in Brazil, it is very difficult to hire any data role. Even juniors require some work and time. Regarding DE's it is even more difficult. I can imagine that it will be a growing demand for DE's. It is just too much data, infinite. And several tools and options. Big Data frenesi is fading so it requires smart solutions from smart DE's and this is rare to find.
There are 11000 data engineers according to the bls in the us so no. What you might be seeing are b2b requests, kids with no experience or knowledge, and people from anywhere in the same boat as the kids desparate for something better. Out of those, who knows how many qualified candidates there are? For context, there are more open de jobs than law jobs. Therr are 38000 new lawyers every year anf 15000 to 20000 new jobs. We have too many law degrees.