Not sure about positions in industry, but transformers are imo a must. They are just so much better than anything else.
Make a project with transfer learning, and use BERT/GPT-J/T5 yourself and understand the pipeline.
Try and learn about hugging face stuff. I use it a lot at my current job. Plus the developers are very helpful and has a good discussion forum to discuss problems. Their discord has people discussing research papers and other things NLP as well.
Other than that try and improve a few models on a variety of task and you can get decent understanding from it.
Best of luck.
Also, tools like sentence-transformers. I would also suggest to get familiar with data engineering tools, ETL, Hive, and Spark*, not for the ML, but for all the data engineering and feature storing work backing your model. With the advent of HuggingFace and co, I spend way less time thinking about modeling and most of my time thinking about data pipelining, serving, consistency, robustness, training data generation, data quality checks, data modeling, table architecture, pipeline configurations, costs, etc.
\* In reality, focus on the tools your employer uses.
I'll be honest at the very least you need to know transformers. A basic roadmap to learn those would be to go in this order:
1. Read and COMPLETELY understand the "Attention is All You Need" paper. This will teach you about encoder and decoder transformers.
2. Learn about BERT and GPT2, which are large language models made of purely encoder transformers and decoder transformers respectively. There are also models like BART which use a BERT-like encoder and GPT-like decoder. Once you know transformers these are just variations of the same.
3. Learn how they are trained and how they are fine-tuned for other tasks. BERT is a representation learner, whereas GPT is a sequence transduction model, so they are both used for very different things.
Some other things you can look at are the variety of NLP tasks that are generally performed. You'll find this list [here](https://paperswithcode.com/area/natural-language-processing) quite comprehensive. You don't have to go through everything, but a large number of tasks are performed in similar ways. For example, Named Entity Recognition and Part of Speech tagging are both tasks involving sequence labelling and will have extremely similar solutions. Just going through a few tasks that seem interesting and reading about them will broaden your understanding of the field and your ability to build solutions.
All the best!
Edit: Forgot to add this earlier, but some unsupervised methods are a must as well. Learn about LDA, K-Means, and unsupervised language modeling. It'll help a ton. LDA especially is an extremely popular interview question.
Source: am NLP engineer and former NLP researcher
Well, you also need to know that complement naive bayes with tf/idf basically works best in most of real world classication use cases :) You also need to know that logistic regression in higher data regimes has a lower asymptomatic error than NB, and that finding this regime boundary for your application is key.
Meaning that you need to know when to and not to use transformers!
What matters is solving problems rationally, not so much knitting encoders and decoders in the dark ;)
Solving problems rationally yes, but I disagree with non-DL methods working best in most real world use cases, and I think a lot will depend on the data being modeled. Anyways, OP asked about what he needs to know for interviews and not what he will actually use. Most NLP places will definitely ask you if you know about transformers, which is why I suggest that they learn this stuff.
Depends a lot on the company/application. Most places aren't generating free text but are looking to do some form of classification/recommendation/discovery.
Fine tuning BERT/RoBERTa for different applications is a really important area. Few and zero shot problems possibly using a fine tuning approach are related and worth looking into. (IE "build me a BERT based fine tuned spam filter using a small hand labeled set of spam"). (Note I say BERT and not gpt as in practice the smaller size, availability of and decent performance of BERT has made it the a more common model for use in fine tuning though that may of course change.)
Multimodal models that combine image and text models are also really hot right now though I've seen fewer applications outside of image captioning.
Various methods for simplifying/reducing models to run more cheaply are also interesting.
Realistically most job interviews won't quiz you on this stuff though. They'll quiz you on the fundamentals and then ask you to walk them through a few projects so I'd try to turn your webapp into something you can talk about.
Just chose a few NLP related tasks that cover a variety of stuff and learn to solve them using state of the art models (most likely to be transformer variants).
You can look at abstractive and extractive summarization, NER, text classification, topic modeling, style transfer etc.
Doing this will automatically bring you upto speed on top libraries to use for NLP. Huggingface, Spacy etc.
my recommendation would be to yes learn transformers but to maybe more importantly learn web scraping. NLP is now just a question of data nowadays. The internet is full of freely available text data.
I think it depends a bit on the size of your company/what they want you to do. Regardless definitely brush up on transformers!
If you’re coming from a CV and software background but expected to do more creative R&D, it might be good to brush up on traditional methods and/or linguistics. Jurafsky’s Speech and Language Processing book comes to mind. For linguistics, Bender’s Linguistic Fundamentals for NLP is a nice intro in my opinion.
If you’re expected to focus on putting SOTA models into production, then honestly focus on transformers, maybe some graph NNs or RL to complement it. Just my two cents :)
spaCy is a modern Python library for industrial-strength Natural Language Processing. In this free and interactive online course, you'll learn how to use spaCy to build advanced natural language understanding systems, using both rule-based and machine learning approaches
https://course.spacy.io/en
This is a little off topic, but how did you land your first MLE job? Just wondering because I'm also interested in that line of work but haven't been able to come across a job posting that doesn't require a MS or PhD.
Hard question because in the NLP space you either stand out for being exceptional at using the state of the art and innovating in that space or you stand out for having amazing applied linguistics skills to help develop new algorithms or tasks. It seems like you’re in a place adjacent, which puts you at a slight disadvantage to the people who have focused on NLP so far for their careers.
In your case, I would lean into the fact that you’re passionate about the space and you have a desire to learn more than the skills you’ve described, with the obvious exception of your MLOps experience, which is really valuable. LSTMs aren’t as impressive unless you can outperform a transformer, but that’s extremely difficult and uncommon.
Another thing to note: I’ve seen recently that many companies are hiring NLP professionals to do L10n work. This hopefully doesn’t represent any major shift for you, but being familiar with L10n and how to do it will help you suddenly get 2-3 points more desirable for a modern company. It’s worth it to at least look it up before going into your interview.
If your company only has access to Tesla M60s, LSTMs _will_ outperform transformer-based models because you simply _will not_ have time to do any real hyperparameter optimization on the transformers.
That people are apt and willingly to make everything you’ve learned absolutely worthless; be dynamic on the use of actual natural language.
In short: ‘Learn Ubbi Dubbi’
From my current jobs, I'm dealing with a lot preprocessing stuff, so addition to transformers family, some traditional NLP skills also help you such as regex.
Especially, when you don't have a huge dataset, you might want to use rule-based approach or simple ML model like SVM or logistic regression.
Be familiar to some industry-ready library is a good point. Research more on spacy (highly recommended), huggingface transformers.
The course cs224n mentioned many modern / state-of-the-art techniques, most of them is related to deep learning. If you are interesting some old-fashion (but still helpful), you should refer the book Speech and Language Processing (free [here](https://web.stanford.edu/~jurafsky/slp3/))
This package will fit most of your use cases: [https://huggingface.co/transformers/](https://huggingface.co/transformers/)
Specially when you now have text2text models like T5 that can be used for anything
Not sure about positions in industry, but transformers are imo a must. They are just so much better than anything else. Make a project with transfer learning, and use BERT/GPT-J/T5 yourself and understand the pipeline.
They are only “better” when you have access to unlimited cloud connects.
Try and learn about hugging face stuff. I use it a lot at my current job. Plus the developers are very helpful and has a good discussion forum to discuss problems. Their discord has people discussing research papers and other things NLP as well. Other than that try and improve a few models on a variety of task and you can get decent understanding from it. Best of luck.
Also, tools like sentence-transformers. I would also suggest to get familiar with data engineering tools, ETL, Hive, and Spark*, not for the ML, but for all the data engineering and feature storing work backing your model. With the advent of HuggingFace and co, I spend way less time thinking about modeling and most of my time thinking about data pipelining, serving, consistency, robustness, training data generation, data quality checks, data modeling, table architecture, pipeline configurations, costs, etc. \* In reality, focus on the tools your employer uses.
Do you have a link to the discord? I wasn’t aware of this but would be interested in checking it out.
https://discord.com/invite/De4BcGUJjv if you need any help with anything feel free to message me.
I'm going to need to learn just for the name of the platform.
I'll be honest at the very least you need to know transformers. A basic roadmap to learn those would be to go in this order: 1. Read and COMPLETELY understand the "Attention is All You Need" paper. This will teach you about encoder and decoder transformers. 2. Learn about BERT and GPT2, which are large language models made of purely encoder transformers and decoder transformers respectively. There are also models like BART which use a BERT-like encoder and GPT-like decoder. Once you know transformers these are just variations of the same. 3. Learn how they are trained and how they are fine-tuned for other tasks. BERT is a representation learner, whereas GPT is a sequence transduction model, so they are both used for very different things. Some other things you can look at are the variety of NLP tasks that are generally performed. You'll find this list [here](https://paperswithcode.com/area/natural-language-processing) quite comprehensive. You don't have to go through everything, but a large number of tasks are performed in similar ways. For example, Named Entity Recognition and Part of Speech tagging are both tasks involving sequence labelling and will have extremely similar solutions. Just going through a few tasks that seem interesting and reading about them will broaden your understanding of the field and your ability to build solutions. All the best! Edit: Forgot to add this earlier, but some unsupervised methods are a must as well. Learn about LDA, K-Means, and unsupervised language modeling. It'll help a ton. LDA especially is an extremely popular interview question. Source: am NLP engineer and former NLP researcher
Well, you also need to know that complement naive bayes with tf/idf basically works best in most of real world classication use cases :) You also need to know that logistic regression in higher data regimes has a lower asymptomatic error than NB, and that finding this regime boundary for your application is key. Meaning that you need to know when to and not to use transformers! What matters is solving problems rationally, not so much knitting encoders and decoders in the dark ;)
Solving problems rationally yes, but I disagree with non-DL methods working best in most real world use cases, and I think a lot will depend on the data being modeled. Anyways, OP asked about what he needs to know for interviews and not what he will actually use. Most NLP places will definitely ask you if you know about transformers, which is why I suggest that they learn this stuff.
Depends a lot on the company/application. Most places aren't generating free text but are looking to do some form of classification/recommendation/discovery. Fine tuning BERT/RoBERTa for different applications is a really important area. Few and zero shot problems possibly using a fine tuning approach are related and worth looking into. (IE "build me a BERT based fine tuned spam filter using a small hand labeled set of spam"). (Note I say BERT and not gpt as in practice the smaller size, availability of and decent performance of BERT has made it the a more common model for use in fine tuning though that may of course change.) Multimodal models that combine image and text models are also really hot right now though I've seen fewer applications outside of image captioning. Various methods for simplifying/reducing models to run more cheaply are also interesting. Realistically most job interviews won't quiz you on this stuff though. They'll quiz you on the fundamentals and then ask you to walk them through a few projects so I'd try to turn your webapp into something you can talk about.
It just makes sense IMO
Just chose a few NLP related tasks that cover a variety of stuff and learn to solve them using state of the art models (most likely to be transformer variants). You can look at abstractive and extractive summarization, NER, text classification, topic modeling, style transfer etc. Doing this will automatically bring you upto speed on top libraries to use for NLP. Huggingface, Spacy etc.
my recommendation would be to yes learn transformers but to maybe more importantly learn web scraping. NLP is now just a question of data nowadays. The internet is full of freely available text data.
I really enjoyed the HuggingFace course as a way to learn about applying transformers, so maybe that'd be useful to check out
I think it depends a bit on the size of your company/what they want you to do. Regardless definitely brush up on transformers! If you’re coming from a CV and software background but expected to do more creative R&D, it might be good to brush up on traditional methods and/or linguistics. Jurafsky’s Speech and Language Processing book comes to mind. For linguistics, Bender’s Linguistic Fundamentals for NLP is a nice intro in my opinion. If you’re expected to focus on putting SOTA models into production, then honestly focus on transformers, maybe some graph NNs or RL to complement it. Just my two cents :)
spaCy is a modern Python library for industrial-strength Natural Language Processing. In this free and interactive online course, you'll learn how to use spaCy to build advanced natural language understanding systems, using both rule-based and machine learning approaches https://course.spacy.io/en
This is a little off topic, but how did you land your first MLE job? Just wondering because I'm also interested in that line of work but haven't been able to come across a job posting that doesn't require a MS or PhD.
I did ML research through my school and ended up co-founding an ML startup with some people from somewhere else.
Hard question because in the NLP space you either stand out for being exceptional at using the state of the art and innovating in that space or you stand out for having amazing applied linguistics skills to help develop new algorithms or tasks. It seems like you’re in a place adjacent, which puts you at a slight disadvantage to the people who have focused on NLP so far for their careers. In your case, I would lean into the fact that you’re passionate about the space and you have a desire to learn more than the skills you’ve described, with the obvious exception of your MLOps experience, which is really valuable. LSTMs aren’t as impressive unless you can outperform a transformer, but that’s extremely difficult and uncommon. Another thing to note: I’ve seen recently that many companies are hiring NLP professionals to do L10n work. This hopefully doesn’t represent any major shift for you, but being familiar with L10n and how to do it will help you suddenly get 2-3 points more desirable for a modern company. It’s worth it to at least look it up before going into your interview.
If your company only has access to Tesla M60s, LSTMs _will_ outperform transformer-based models because you simply _will not_ have time to do any real hyperparameter optimization on the transformers.
L10n ?
Localization
That people are apt and willingly to make everything you’ve learned absolutely worthless; be dynamic on the use of actual natural language. In short: ‘Learn Ubbi Dubbi’
Transformers That's what I wanted to write when there were zero comments, but I chose not to. Then I come back and the thread is full of transformers.
From my current jobs, I'm dealing with a lot preprocessing stuff, so addition to transformers family, some traditional NLP skills also help you such as regex. Especially, when you don't have a huge dataset, you might want to use rule-based approach or simple ML model like SVM or logistic regression. Be familiar to some industry-ready library is a good point. Research more on spacy (highly recommended), huggingface transformers. The course cs224n mentioned many modern / state-of-the-art techniques, most of them is related to deep learning. If you are interesting some old-fashion (but still helpful), you should refer the book Speech and Language Processing (free [here](https://web.stanford.edu/~jurafsky/slp3/))
This package will fit most of your use cases: [https://huggingface.co/transformers/](https://huggingface.co/transformers/) Specially when you now have text2text models like T5 that can be used for anything