MageLab

Mage Lab Documentation

Mage Lab Documentation

Mage Lab Beta

Welcome! Mage Lab is a desktop application that provides an AI interface for you and your computer. By simply talking to Mage, you can open files, browse the web, play or view media like audio, images, or videos, edit documents, or even create entire projects. Mage can also read websites and files (upon request) . This all comes out of the box when using supported language models.

The core functionality of Mage Lab will be sufficient for many use cases. From there, you can easily customize its abilities by scoping which 'tools' it has access to in the settings panel. This is where the power of Mage Lab becomes self evident, having the ability to whimsically modify or expand Mage's capabilities on the fly.

We designed this software for our laboratory so we could control our equipment and data in house. Mage Lab boasts a very strong and cost effective performance with supported open source and proprietary AI models alike. Mage Lab prioritizes privacy, transparency, and simplicity.

Mage Lab can use AI to reason through complex tasks while using tools at the same time, which is incredibly powerful. These abilities will continue to scale and impress as LLMs mature. Additionally Mage Lab utilizes voice integration as a first class citizen in the interface. Talking or typing are both part of the same chat flow.

Introduction

Mage Lab serves as an interoperability layer, providing an interactive user interface that listens to your requests and enhances your workflow with your preferred AI tools. Core functionality includes:

  • Searching the web
  • Opening links and files
  • Writing new files
  • Reading websites and files
  • Searching for folders and files
  • Math calculations
  • Navigating directories

Mage will easily use these tools to help you with what you are working on, sometimes without even asking for them directly. That said, Mage only knows what you tell it, and can only use tools that are enabled in the settings. If for example, you don't want Mage to be able to read the files on your computer, you can simply toggle it off. Likewise, tools that are not toggled on will not be available for use during that session.

Installation

Installation made easy!

Consumer Path with Installer
Developer Path <-- links down to "developer documentation"

Steps to Install Mage Lab

  1. Get the Mage Lab Repository: Clone or download the repository.
  2. Open Your Terminal: Go to the project’s root directory. Usually you can right click the folder and 'Open in terminal' or 'Open with Git Bash here'.
  3. Launch Mage Lab: Run the following script (for Windows use Git Bash):
./launch.sh

After setup is complete your browser should pop open to http://localhost:8001 , or you can go there directly to access the Mage Lab interface.

Quick Start Guide

Please consider Mage Lab as your one easy studio for interacting with AI tools, naturally using both voice and text, while accessing computer features together. You can even interrupt a chat, just like in a human conversation!

Chatting with Mage: Text & Voice

  • Unified Inputs: You can talk to Mage and/or type your commands interchangeably. To use your voice, simply hold the "Hold to Talk" 🎙️ button. Mage will listen and only transcribe and respond when you release the button.

Hands-Free Communication

  • Voice Activity Detection: Experience hands-free communication with Mage! Toggle voice activity detection using 🔴 for a seamless interaction. Mage will listen continuously and will only transcribe and respond after detecting a pause in your speech.

Interrupting / Mute

You can interrupt Mage by clicking "Hold to Talk" 🎙️ button once to stop its speech or holding the button to interrupt and send a new message. To mute Mage speaking entirely, press the mute 🔇 toggle to switch off AI speech in the header bar. You can still talk but Mage will only respond with text.

AI Response Time

Response times can vary depending on the LLM or setup you are using and the complexity of the task. Using custom tools with their own limitations can effect the time it takes for task completion. See Pam(link) for asyncronous task completion

UI Capabilities

This section is designed to familiarize you with the user interface and help you make the most of Mage Lab's interactive features.

Core Features

Mage Lab allows you to open various file types as tabs within its user interface (UI). You can open these files manually with the file picker or just ask Mage to do it!

  • Text Editor: Create and edit documents with ease.
  • Image Gallery: View and manage images directly within the interface.
  • Video Player: Watch videos without leaving the application.
  • Audio Player: Listen to audio files seamlessly.

You can easily open files, write files, and view media in the UI. Mage can help you open links and browse and read the the web right from Mage Lab without lifting a finger.

Core Concepts

Talking to Mage

Asking Mage to do things can be simple and direct. If you know Mage's capabilities you can as for them directly. For example if play audio is toggled on, you can ask "Mage, please play Beethoven" and if you have Beethoven in your Music Path folder it will open an audio player tab and play it for you. Mage does not explicitly "know" about the tabs. The tabs open incidentally, based on file type and activity.

Mage's Tabs

If Mage writes a file, Mage will display it for you in a new tab in the UI, letting you track changes in real time and helping you maintain control over your work. While Mage serves as a supportive tool, it grounds the experience so that you are always in charge of your tasks and decisions!

File Navigation

Mage has built-in file management features that help you stay focused on your tasks by assisting you in locating folders and files directly within the application. Mage's default working directory starts in your Workspace Path. This will be the default location where it will search for and write files. Other features like playing audio default to your Music Path. You can easily update your file paths in configuration in the settings panel to customize how Mage operates. For example, if you want to work on a specific project folder, you can ask Mage to switch to the new project directory so the new files it reads and writes will default to that directory. This keeps everything organized and minimizes the distraction of navigating multiple desktop applications.

Media Types

Mage allows you to open a variety of files directly in the Mage Lab UI, including:

Text file types
  • txt, md, sh, py, js, ino, log, json, csv, html, pdf
Audio file types
  • mp3, wav, flac
Image file types
  • png, jpg, webp, gif, bmp, svg
Video file types
  • mp4

You can open tabs automatically by asking Mage, or manually using the file explorer 📁 (in the header bar) for added convenience.

UI Canvas Elements

  • Home
  • File Browser
  • Close All Closes all open tabs
  • Dark/Light mode
  • Vertical / Side by side
  • Mute / Volume
  • New Chat
  • Configuration Panel

Tools

Mage accomplishes task execution through the use of tools. Tool use in Mage Lab is explicit and all tools are available for configuration in the settings panel. You can see exactly what they do there and which ones are active. Ideally, tool use should be 'invisible' where it uses it incidentally, for example if you ask Mage to "read a file" it should just do it without having to say "use the read file tool". Sometimes you can give Mage a nudge if you have to. Results will vary between LLMs.

Mage Lab can use tools that might not provide visual feedback in the UI. Generally speaking the tabs are for opening various files and links. To integrate opening a tab into your own custom tool review the Developer Documentation.

All tools are clearly visible and completely under your control in the configuration settings. This means you can easily customize which tools you want to use, ensuring that your workspace is tailored to your preferences. With this clear visibility and control, you can enhance your productivity without unnecessary distractions by controlling the tool scope.

Advanced Usage

For those who are already familiar with the basic chat and core functionality in Mage Lab and are looking to enhance their experience and customize the capabilities of the platform, here you'll find details about configuring settings, selecting voice and language models, managing context length, and utilizing various tools. Let's dive into the configuration options and tools available to elevate your experience!

Configuration

Options

Voice Models
  • You can choose from different voice models for your text-to-speech interactions. The availability of certain voice models may depend on your AI provider.
LLM Models
  • Select your preferred language model for chatting. Available models vary depending on what Provider you have selected. We recommend using tool-capable models like gpt-4o-mini and qwen2.5 to get the best experience with Mage Lab.
  • Keep in mind that how well local models perform depends on your computer's specifications.
  • If you choose third-party AI, such as OpenAI, you'll need to have a funded account and an API key.
Context Length
  • For inference providers like OpenAI, context length typically isn't a significant issue. However, remember that more tokens (word segments) both in and out the more it will increase costs.
  • Context length can have a big impact on local LLM performance - even for frontier models. This is especially true with locally hosted models, depending on your computer's specification. This also includes having too much context in the system prompt, not just in the overall chat history.
  • To help manage this, Mage Lab allows you to shorten chat history and adjust your tool set, as well as edit the system prompt. This flexibility lets you tailor performance and behavior to best fit your budget and needs.

Tools

Concept
  • Tools are essential to the Mage Lab interface. Without them, it is just be a simple chat interface. All the advanced features of Mage Lab rely on these tools. While many language models support tool usage, not all do. We suggest using gpt-4o, gpt-4o-mini, gpt-4-turbo, gpt-4.5 or any of the qwen2.5 series models for the best results.
  • Mage Lab includes a basic set of tools that let you open various types of files and media. These tools may or may not provide visual feedback when using them. When you ask Mage to write a file, the file is already written by the time you see the tab open. The tab is incidental, and there to provide a visual queue and additional interactivity for your collaboration.
  • Tools can potentially run in the background without interacting with the interface. However, you can still see the tool calls and tool responses by toggling on tool debugging in the settings panel. This allows you to evaluate the AI responses to see if it is responding honestly, or hallucinating.

Mage Lab is highly customizable, providing endless possibilities for integration and expansion. You can create or import your new tools and the possibilities are endless. See Developer Documentation.

Tool Toggles
  • The tools available in Mage Lab can be configured in the settings panel. Only the tools that are toggled on will be "visible" to Mage. If a tool isn’t toggled on, Mage cannot utilize it, as it is not included in the scope of your current session.

Additional Info

Streaming vs. Tools

Streaming can reduce latency but may not be available with some tool calling models.

  • Streaming: Responses are delivered as streaming text, making the interaction feel smoother and natural—ideal for when you want quick feedback.
  • Tools: Not all models support streaming with tool call, so streaming may not be available.

Models with Tool Support

Tools are central to many features in Mage Lab when considering which models to use. Mage Lab, as a desktop interface, currently works well with the following open-source models that provide tools to support our features: qwen2.5 and llama3-groq-tool-use. Additionally, several OpenAI models, such as gpt-4o and gpt-4o-mini, excel at tool usage and are budget-friendly (an OpenAI API key is required to access them). While you can use other models, their functionality may be limited.

AI Inference Costs

While each AI provider (OpenAI, Grok, etc) offers their own interface for using their models, they also provide API keys to provide AI inference (at a cost). Mage Lab is compatible with all "OpenAI standard" APIs which covers most providers (Anthropic coming soon) . Open source models like qwen2.5 and cheaper proprietary models like gpt-4o-mini offer economical yet impressive performance. For more advanced use cases using frontier level models gpt-4o, gpt-4-turbo, and gpt-4.5 can deliver stunning results. Please note that these commercial models require an API key and usually charge by the token. Mage Lab serves as your user-friendly interface to interact with these various models, allowing you to easily find the perfect fit for your projects! To read more about API keys please refer to the AI provider section.

Power Users

Here, you'll find information on AI providers, model locations, self-hosted options, and custom tools that will empower you to enhance your workflows and maximize the capabilities of Mage Lab.

Customization

AI Providers

Inference providers (e.g. Mage Lab, OpenAI, Grok, Anthropic, etc.) provide not only language model responses, but also transcription, text-to-speech, and additional services. To use a provider, you need the resource's URL (e.g., https://api.openai.com/v1 or localhost:11434/v1) and an API key (when required).

Available AI Providers
  • OpenAI
  • Grok
  • Groq
  • Lambda
  • Ollama (locally hosted)
  • Any OpenAI compatible API
Data Privacy

Mage Lab respects your data privacy and freedom. We maintain no tracking; however, if you send requests to a third party, privacy risks are yours to manage.

In Mage Lab, you can easily select which AI provider fits your needs, whether you’re using cloud services or locally hosted. The available models are linked to the provider, so changing the provider updates what models are available.

Model Types

You can select a separate provider (if desired) for each of the following AI resources in the settings panel:

  • LLM Inference
  • Text to Speech
  • Transcription
  • Vision
  • Image Generation

You can easily add new custom providers in addition to commons providers included in the settings.

Self Hosted (Local AI)

  • Docker: Docker containers are available for TTS and transcription, as well as Ollama.
  • Ollama: Ideal for running LLMs locally within your system capabilities.
  • Recommended Models: Mage Lab suggests qwen2.5 when using locally hosted LLMs.
  • Compatibility: We use the OpenAI standard libraries for responses, transcription, and text-to-speech. Local AI should be OpenAI compatible for use with Mage Lab (like Ollama).

Custom Tools

  • How to Create a Custom Tool
  • Mage currently supports adding tools using properly formatted python functions
  • Decorators are required to to define how Mage will use your custom tool. Ensure names and parameters match your function. You can drop the custom tool file in the backend/functions folder to add multiple tools to one file.
  • UI Functionality: Use the open_tabs(<filepath or url>) function in your custom tool to create tabs.
  • Guidance and Principles: Keep instructions clear and semantic; the LLM does not need to understand the internal workings, just how to use the tool.
  • Installing New Packages: New packages can be integrated into the Python environment from the backend folder using uv add <package name>.
  • Developer Workflow: To use the developer installation, run ./build_launch.sh from the project root.

Developer Documentation

Installation Guide

Installation made easy!

Before You Start

Make sure you have these tools ready:

Steps to Install Mage Lab

  1. Get the Mage Lab Repository: Clone or download the repository.
  2. Open Your Terminal: Go to the project’s root directory.
  3. Launch Mage Lab: Run the following script (for Windows use Git Bash):
./launch.sh

This will:

  • Verify installation.
  • Install uv (a Python environment manager) using pip.
  • Build the frontend components (located in frontend/).
  • Create a virtual environment and synchronize Python dependencies in backend/.
  • Launch Mage Lab.

Manual Installation

If you prefer to install step-by-step (or if you're on Windows without a POSIX-like environment), follow these instructions:

Frontend

Step 1. Navigate to the frontend directory

cd frontend
npm i
npm run build

Step 2. Return to the project root once completed.

Backend

Step 1. Ensure uv is installed:

pip install uv

Step 2. Navigate to the backend directory:

cd backend

Step 3. Create a virtual environment with uv:

uv venv

Step 4. Sync dependencies from pyproject.toml:

uv pip sync pyproject.toml

Step 5. Alternatively, you can utilize backend/docs/requirements.txt with another package manager (pip, conda, etc.).

Manual Usage (Advanced)

To start the backend directly after manually installing dependencies:

cd backend
python -m uv run uvicorn main:app --reload --host http://localhost --port 8001 --log-level critical

Then, open your browser to http://localhost:8001 to access the Mage Lab interface.