What is Petals?
Petals lets people run and fine-tune large language models by connecting a computer that holds part of a model to a community network serving the remaining parts. It is aimed at developers and researchers who want to work with models such as Llama, Mixtral, Falcon, and BLOOM from a desktop computer or Google Colab. The tool provides an API-style way to run models while retaining access to PyTorch and Transformers workflows.
Petals Features:
Distributed Models: Connect to community-hosted model layers to generate text with models including Llama 3.1, Mixtral, Falcon, and BLOOM.
Local Inference: Load a portion of a model on a consumer-grade GPU or in Google Colab, then send requests through the distributed network.
Model Fine-Tuning: Fine-tune supported large models for specific tasks, with tutorials covering prompt tuning for classification and chatbot creation.
Sampling Methods: Apply fine-tuning and sampling methods to model runs rather than being limited to a basic text-generation API.
Custom Model Paths: Run custom paths through a model and inspect its hidden states for more detailed experimentation.
GPU Hosting: Contribute GPU capacity by hosting model layers, or create a private swarm for a group of trusted participants.
PyTorch Compatibility: Work with models using the flexibility of PyTorch and Hugging Face Transformers alongside an API-style interface.
Petals Benefits:
Lower Hardware Bar: You can explore models too large to load fully on one consumer computer by sharing the work across network participants.
More Experimentation: Access to custom paths, sampling methods, and hidden states makes it easier to investigate model behavior beyond generated answers.
Task Adaptation: Fine-tuning lets you shape a supported model around a particular task instead of relying only on its general-purpose behavior.
Flexible Workflow: The PyTorch and Transformers interface supports research and development workflows that need more control than a standard API call.
Shared Capacity: People with available GPUs can contribute to the network, helping make hosted model capacity a collaborative resource.
Use Cases:
Interactive Chatbots: Build or test a chatbot that generates replies using a large model served across the Petals network.
Prompt Tuning: Adapt a model for semantic text classification or a personified chatbot using the prompt-tuning workflows.
Model Research: Inspect hidden states or try custom model paths when studying how a large language model processes inputs.
Task Fine-Tuning: Tune a supported model for a domain-specific task when a general model’s outputs need to be adapted.
Shared GPU Hosting: Contribute a GPU to serve part of a model, or set up a private swarm for collaborators who trust one another.