What is selfgpt?
selfgpt is an experiment for testing whether language models can generate user-side chat messages, rather than only assistant replies. It compares two ways of producing those messages: reversing a chat template so the model continues in the user slot, and prompting it to roleplay a human. A judge compares generated replies with real messages from ShareChat, and an interactive results page lets visitors inspect model performance and try judging a pair themselves.
selfgpt Features:
Native Mode: This configuration reverses the chat template and lets the model generate a message in the user slot without fine-tuning.
Roleplay Mode: This configuration asks the model to act human when generating a user-side message.
Pairwise Judging: A judge sees the conversation so far and both a real reply and an AI-generated reply, then identifies which one came from a person.
Order Checks: Each reply pair is presented in both orders so the evaluation tests both arrangements.
Model Scores: A model's score reflects how often it fools the judge, with 50 percent meaning the judge cannot tell the replies apart.
Results Explorer: The results display uses shaded squares for conversations, and visitors can click a square to see its details.
Judging Test: A quick interactive test lets visitors try deciding which of two messages was written by a person.
selfgpt Benefits:
Compare Approaches: Seeing native and roleplay results side by side helps clarify how the generation method affects the messages.
Ground Model Choices: The judging scores offer a consistent basis for comparing models instead of relying only on impressions of their replies.
Spot Repetition: Inspecting conversations makes it easier to notice whether generated openings vary or recycle a small set of patterns.
Question Model Size: The results help show that a larger model is not necessarily more convincing at producing user-side messages.
See Evaluation Limits: The project notes that the judge's lack of logprob support makes the results less precise than they could be.
Use Cases:
Test User Simulation: Evaluate whether a language model can produce plausible user-side messages in an existing conversation.
Compare Prompting Methods: Compare reversed-template generation with a direct instruction to roleplay a human.
Rank Candidate Models: Use the reported judge scores to compare models on how often their generated messages are mistaken for real ones.
Review Conversation Openings: Inspect generated openings to see whether a model varies how it starts conversations or repeats a few patterns.
Try Human Judging: Use the interactive test to practice distinguishing a real message from an AI-generated one in context.

