Perplexity’s mission is to democratize access to knowledge and transform how people discover information. Perplexity faced a significant technical challenge: implementing secure, scalable code execution within their search results. The code interpreting features are now currently available to all Pro users.

The challenge of running code

Many times when people are asking questions, there is an underlying need to run code. If I’m looking at a stock price, I want to see a chart with its price trends. When searching for election data, it’s helpful to see a breakdown of votes for different candidates. It’s fairly easy for the models to generate the code today, but when you are serving millions of users like Perplexity is, there are real infrastructure challenges to supporting code interpreter capabilities:

Security: It's not safe to let the LLM run code and use tools in the same environment where your application is running. Isolation of executed code to prevent unauthorized access to system resources or user data.

Performance: Users have a short attention span, and you need to provide answers quickly. LLMs are already slow enough at generating tokens, that everything else needs to be tuned to the max.

Scalability: Spinning up thousands of coding runtimes in parallel is not trivial and can cause degraded performance.

Secure code execution at scale

Perplexity has 340 million searches per month so they needed a solution that works at scale. That was part of the reason why Perplexity ultimately chose E2B Sandboxes for running the underlying code. 

Key factors in this decision included:

• Security and isolation: E2B's use of Firecrackers by AWS for creating isolated environments is a secure way to run LLM-generated code.

• Ease of use: E2B comes with a built-in Python and Javascript SDK for easy development

• Stateful, highly customizable code execution: E2B sandboxes are able to do almost anything you want them to do. They support stateful code execution, are super performant, and come with highly adjustable settings that you can tune to suit your needs.

The E2B's sandboxes are isolated cloud environments made specifically for AI code interpreting or code execution. When a new sandbox session is started, E2B starts a small VM in the cloud. All this takes about 150-170 ms.

Inside the E2B Sandbox, Perplexity can run code generated by the LLM, start any programs, use the filesystem, start long-running processes, upload data to the Sandbox, and download any type of file from the Sandbox.

“We are now running millions of E2B Sandboxes each month”

- Denis Yarats, CTO of Perplexity

Implementing E2B

Perplexity took the first steps to implement E2B cloud runtime in July 2024 and the first version of the code interpreter was in production in August. The integration required close collaboration with the E2B engineering team.

The most important steps were:

• Establishing requirements: For example a decision was made to use stateful code execution, instead of stateless.

• Infrastructure scaling: It was important to scale up infrastructure for enterprise-grade use and plan the load in advance to be prepared for the volume of Perplexity search queries.

• Collaborating on features: Perplexity and E2B worked together on sandbox features (e.g. matplotlib data extraction) to match some of the UX nuances of Perplexity.

“We had E2B up and running within a week.”

- Denis Yarats, CTO of Perplexity

What’s next

Since the implementation of the E2B code execution layer, Perplexity observed significant growth in engagement with coding-related queries. Additionally, interactive charts have wowed users. Together with E2B, Perplexity designed a new feature that allows developers to customize charts and make them interactive on the front end.