AI Engineers: A new Field

We're on the verge of a significant change in the field of artificial intelligence (AI): the emergence of the AI Engineer. This new role within software development will be crucial in harnessing the potential of AI technologies and making them useful in the real world.

The Increase of AI Use

AI technology is becoming more common, thanks to the accessibility of open-source APIs and advancements in large language models. What used to take a lot of time and knowledge can now be done much faster with the right tools.

Andrej Karpathy, head of AI at Tesla, believes that there will soon be more AI Engineers than machine learning (ML) engineers. Interestingly, someone can excel as an AI Engineer without direct training experience.

The Challenges for AI Engineers

There are several challenges that AI Engineers will need to overcome:

• Evaluating different models, from the largest GPT-4 and Claude models to smaller open-source ones.
• Making the most of popular tools, such as LangChain, LlamaIndex, and Pinecone, and new autonomous agents like Auto-GPT and BabyAGI.
• Keeping up with a growing amount of information, including papers, models, and techniques.

To meet these challenges, software engineering is likely to evolve a new specialty: the AI Engineer. These professionals will focus on using the AI stack effectively.

The AI Engineer Revolution

In tech start-ups, what used to be informal groups chatting about AI are now becoming formal teams. The term 'AI Engineer' is increasingly being used for professionals who make AI APIs and open-source models useful in products.

AI Engineers are making a big impact in large tech companies like Microsoft and Google, innovative start-ups, and even among independent developers. They're transforming product development by using AI advancements to create user-friendly products quickly.

One key point: you don't need to have an academic background to be successful in shipping AI products. Engineers are more important than researchers.

The Shift from ML to AI Engineering

Right now, there are more ML Engineer jobs than AI Engineer roles. However, the rapid growth of AI suggests that this will change within the next five years.

The common perception is that AI Engineering is similar to Machine Learning or Data Engineering. But many successful AI Engineers haven't had traditional Data Science/ML training. This highlights a significant change in the learning path needed to become an AI Engineer.

Why AI Engineers Are Emerging Now

Several factors are contributing to the rise of AI Engineers:

• The growth of Foundation Models, known as "few-shot learners," which can learn in-context and transfer knowledge better than intended by their original trainers.
• Large companies like Microsoft, Google, and Meta offering "AI Research as a Service" APIs, making AI capabilities available even without hiring research talent.
• More and more start-ups investing heavily in their own hardware, attracting considerable seed funding and creating more opportunities for AI Engineers.
• AI's ability to quickly validate product ideas, greatly reducing the cost of AI product validation.
• The expansion of AI tools for the JavaScript developer audience.
• The shift from "classifier ML" to "generative AI," which is key for developing innovative AI applications.
• The important role of human-written code in the shift from Software 2.0 to Software 3.0, marking a move from 'software atop intelligence' to 'intelligent software'.

The Future of AI

The growing number of AI Engineers signals a new era of practical optimism in technology. This new breed of professionals will not only use AI, but shape it, working towards a future where the line between human and AI engineering may no longer be clear.

‍