Today, Muna supports running Google’s Embedding Gemma text embedding model locally and in the cloud—by changing only one line of code.

Embedding Gemma has shown very strong performance across a range of tasks, including question answering, clustering, and summarization. Along with it, we are launching our OpenAI client in our Python and JavaScript libraries.

From OpenAI to Open Source

While designing an interface for developers to use Embedding Gemma, we were faced with a stark reality: the overwhelming majority of developers use text embedding models via OpenAI’s client libraries.

To make the transition as smooth and painless as possible, we are shipping an OpenAI client in our SDKs that will allow engineering teams run open-source text embedding models with almost the exact same code.

Thanks to Muna’s compiler technology, the embedding models will be downloaded and run locally by default. That said, developers can run the models on cloud GPUs with only one additional line of code:

If you already have compute resources (VPC, bare metal), reach out to us for how to run your compiled embedding models on your own infrastructure.

Bring Your Own Embedding Model

You can compile any arbitrary Python function to be compatible with our OpenAI client’s embedding interface, with just some type hints:

With this, engineering teams can compile their own custom or fine-tuned models with Muna, and run them anywhere. The Embedding Gemma model itself was compiled with under 100 lines of Python code.

If you build something fun with this, tag us—we’ll feature it in subsequent blog posts. And if you have any questions,

Join us on Slack

Realtime Detection Transformer is a leading object detection model, beating YOLO on object detection tasks thanks to its transformer architecture. Normally to deploy this model, developers must first export the model weights from PyTorch to a chosen inference format (e.g. ONNX, TensorRT), then write C++ code to run the model in their target environment.

Compiling the Model with Muna

Muna completely automates the process of writing deployment-ready C++ code for model inference. First, we wrote a Python function that takes an image, runs the model, and returns a set of Detection objects. All of this fit neatly into just 150 lines of Python code. From there, we compiled the function1 with Muna:

Once compiled, anyone can run the model from Python, JavaScript (browser and Node.js), Swift (iOS), Kotlin (Android), and so on. Here’s an example showing how to run the model in JavaScript (generate an access key for free here):

At runtime, the compiled model will get downloaded from Muna, loaded into your application, and then invoked with your input image. Don’t take our word for it, try the online demo yourself!

Try RT-DETR on Muna

If you have any questions, come bother us on our community Slack. Happy coding!

Over the past few weeks, we’ve been working to add support for LLM inference to Muna. Muna allows you to write Python functions, and compile them to run anywhere in as little as two lines of code:

And since everyone is used to calling openai.chat.completions.create(…) in their code, we decided to implement a similar, experimental API in our client libraries:

We’ve got this implemented in our Python, JavaScript (browser + Node.js), and Android client libraries, with the same API coming in our Swift, React Native, and Unity Engine client libraries. Come talk to us about it in our Slack community.

Try out this demo project, which runs Google’s Gemma 3 270M using the llama-cpp-python library in Python; along with a Next.js web app that runs the compiled LLM locally in the browser.

Happy AI-ing.

Subscribe now