Continuous companion loop
Listening, barge-in, actions, suggestions, risk interruption, and spoken follow-up remain coordinated.
v0.11.1 Adaptive Multimodal Companion
The Open-Source AI System Control Agent
Control your entire computer with natural language, voice, and hand gestures. An open-source, privacy-first AI agent that plans, executes, and verifies complex multi-step tasks — running locally on your hardware.
156
Actions
30+
Gestures
21
Agents
Tier 5
Security
Scroll to call the agents
Intuitive Command. Infinite Power.
Control your entire digital ecosystem with zero friction through four core modalities.
mic
Voice & Speech
"Hey Heliox" ambient wake-word with VAD-based endpoints, barge-in support, and real-time push-free dispatch for frictionless execution.
front_hand
30+ Hand Gestures
Static poses (Palm, Pinch) and motion gestures (Two-Finger Swipe) with kinematic smoothing, 3D world-model, and optional gaze tracking fusion.
visibility
Always-On Awareness
Continuous computer-vision loop taking screenshots every 3s, detecting active apps, and maintaining a rolling context buffer fed into the LLM planner.
bolt
Autonomous Jobs
Fire-and-forget background tasks that decompose, execute, and verify completely independent of the UI or main event loop.
psychology v0.11.1 — Adaptive Multi-Agent
JARVIS Autonomy
Heliox OS combines reactive commands with opt-in proactive and background capabilities — all running through the same permission and verification pipeline.
Brain Load: 84%
arrow_right_alt
Cognitive Offload
🧠
Adaptive Proactive Suggestions
Pattern-matches local screen context and surfaces visible, optional help before you ask. Accept/dismiss feedback tunes each pattern's timing and priority, and temporarily suppresses repeatedly rejected suggestions.
🤝
Interactive Execution Companion
Independently reviews proposed plans, can warn, revise, or stop work that drifts from the request, accepts typed or spoken corrections while a task is running, and offers grounded next ideas after verified results.
⚡
Fire-and-Forget Jobs
Spawn complex multi-step background tasks that decompose, execute, and verify completely independent of the UI or main event loop.
👁️
Always-On Screen Awareness
Auto-bootstrapped computer vision tracks your contextual state cross-platform, natively bridging exactly what you see into the LLM planner.
🎤
Continuous Voice Listener
"Hey Heliox" ambient wake-word with VAD-based speech endpoints and barge-in — start talking and Heliox stops mid-sentence to listen.
🤚
30+ Hand Gestures & Air Drawing
Static poses (Palm, Pinch) and motion gestures (Two-Finger Swipe) with kinematic smoothing. Opt-in 3D world-model via MediaPipe HandLandmarker and coarse gaze-tracking fusion.
🖱️
Gesture Cursor Control
Point to move the real OS cursor, pinch to click — opt in via Settings. Open palm always exits instantly. Off by default for safety.
🎯
Adaptive Calibration
On-device continual-learning loop personalizes pinch/thumb thresholds and wake-word matching from implicit signals — no retraining, bounded, resettable.
🌀
Arc Reactor UI & Ambient HUD
Animated, immersive Tauri overlays responding contextually to system actions with real-time visual feedback and status indicators.
neurology Zero-Dependency · On-Device
Cognitive Engine
A lightweight, dependency-free engine estimating attention, stress, and cognitive load from local interaction-history heuristics. Zero external models, no network, no license restrictions.
monitor_heart
Cognitive HUD
Real-time attention/stress/load estimates mapped onto the Svelte UI.
record_voice_over
Dynamic TTS Pacing
JARVIS automatically slows voice generation during high cognitive-load tasks.
shield
Stress-Aware Gate
High-risk actions evaluate cognitive stress first. If distracted, a 10-second auditory confirmation gate holds.
fingerprint
Persona Fingerprint
Subconscious background loop learns cognitive engagement patterns and encodes them to persona.md.
notifications
Smart Notifications
Notification pipeline buffers trivial alerts until a low-load resting state is detected.
speed
ReAct Cost Estimator
Tasks predict aggregate cognitive demand. If executing risks exceeding mental bandwidth, JARVIS pauses.
auto_awesome
JARVIS Intent Classifier
Fully IntegratedNative intent fusion classifying spoken commands against current workload intensity for context-aware execution routing.
Modular Cognitive Architecture
A true agentic system running a continuous ReAct loop with a modular multi-agent orchestrator — not a simple command runner.
01
Gateway Hub
LLM evaluates persistent memory context before reasoning
02
Planner
Converts natural language into structured multi-step action plan
03
Orchestrator
Routes each action to 21 specialists across 20 source-scoped roles
04
Specialists
21 domain specialists execute through reviewed OS, browser, and integration adapters
05
Verifier
Post-execution verification confirms action success
06
Reflector
Self-improvement engine learns from successes and failures
07
Security
Five-tier permission system with confirmation gates & rollback
21 Specialist Agents
terminal
System
folder
File Ops
inventory_2
Package
settings
Service
desktop_windows
Desktop
code
Code
merge
Git
account_tree
Workflow
language
Web
wifi
Network
hub
Integration
extension
Plugin
mail
Comm
forward_to_inbox
Email
calendar_month
Calendar
rss_feed
RSS
visibility
Vision
monitoring
Monitor
search
Forensics
manage_search
Semantic
laptop
SSH
account_tree
Task Decomposition
Complex goals auto-broken into dependency-aware subtask trees with parallel execution paths.
science
Simulation Sandbox
Pre-execution impact reports for dangerous commands showing risk level, affected files, and reversibility.
trending_up
Self-Improving Prompts
Successful reasoning chains stored and reused with keyword-indexed templates and success/failure rates.
labs
Evolution Harness
Detached worktrees evaluated in a network-disabled, credential-free Docker runner — no automatic code promotion.
156 Action Types
The planner schema declares 156 action types with 156/156 provider coverage. Availability depends on OS, dependencies, credentials, and security policy.
folder File Operations
file_readfile_writefile_deletefile_movefile_copyfile_listfile_searchdirectory_summarydirectory_sizefile_hashfile_comparefile_permissions
memory Process Management
process_listprocess_killprocess_info
terminal Shell & Code
shell_commandshell_scriptcode_execute
language Browser & Web
browser_navigatebrowser_clickbrowser_typebrowser_extractbrowser_extract_tablebrowser_extract_links
screenshot_monitor Screen & Vision
screenshotscreen_ocrscreen_analyzescreen_detect_elements
inventory_2 Package Management
package_installpackage_removepackage_updatepackage_search
info System Info
system_infocpu_usagememory_usagedisk_usagenetwork_infobattery_info
web_asset Window Management
window_listwindow_focuswindow_closewindow_minimizewindow_maximize
merge Git
git_statusgit_diffgit_loggit_branchgit_stagegit_commitgit_pushgit_resolve
extension Plugins · WASM · Skills
plugin_callwasm_callskill_run
power_settings_new Power Management
power_shutdownpower_restartpower_sleeppower_lockpower_logout
volume_up Audio / Volume
volume_getvolume_setvolume_mute
brightness_6 Display / Screen
brightness_getbrightness_setscreenshot
wifi Network / WiFi
wifi_listwifi_connectwifi_disconnect
content_paste Clipboard
clipboard_readclipboard_write
schedule Scheduled Tasks
schedule_createschedule_listschedule_deletetrigger_create
data_object Environment Variables
env_getenv_setenv_list
dns Service Mgmt (Linux)
service_startservice_stopservice_restartservice_enableservice_disableservice_status
devices Platform-Specific
gnome_setting_readgnome_setting_writedbus_callregistry_readregistry_write
open_in_new Open · Launch · Notify
open_urlopen_applicationnotifydownload_file
Plugin Ecosystem
Two extension paths: auto-discovered local plugins with Ed25519 signature verification, and a reviewed GitHub marketplace with CI-enforced moderation.
cloud
weather
MarketplaceGet real-time weather data for any location via voice or text commands.
music_note
spotify-control
MarketplaceControl Spotify playback, search tracks, and manage playlists via gestures or voice.
home
home-assistant
MarketplaceBridge your smart home devices into the Heliox agent for unified automation.
verified
Ed25519 Signature Verification
Every plugin is signed. Unsigned, untrusted, or tampered packages are rejected before their manifest or code is loaded. Planner-triggered plugin actions route through the normal permission system.
Built for Privacy & Security
Run entirely on local hardware for total privacy, or seamlessly connect to frontier cloud models. All AI outputs pass through structured schema validation before execution.
Hybrid
Local & Cloud LLMs
AES-256
Vault Encryption
HMAC
Audit Chain
shield
Five-Tier Permission System
Read-only through root-level with confirmation gates and per-action granular approve/deny.
science
Simulation Sandbox
Pre-execution impact reports for dangerous commands with risk assessment and reversibility tracking.
psychology
LLM Safety Critic
Secondary LLM independently reviews Tier 3-4 plans before the confirmation gate fires.
model_training
Learned Risk World Model
Bundled MLP predicts action impact from training samples. Can interrupt but never remove rule-based warnings.
history
Snapshot Rollback
Btrfs/Timeshift on Linux, Windows Restore Points. If backend unavailable, destructive action doesn't run.
record_voice_over
Pocket TTS (Open-Source Voice)
Free, CPU-only local speech model — no API key, no cost. Falls back to platform TTS if not installed.
stop_circle
Mid-Flight Cancellation
Stop button genuinely kills in-flight subprocess (proc.kill()) instead of only stopping the next action.
key
Encrypted Key Storage
API keys stored via platform keyring (GNOME Keyring / Windows Credential Manager). Never logged or sent to local LLMs.
new_releases
Public release · v0.11.1
Adaptive Multimodal Companion
The largest Heliox release yet: coordinated always-on voice, bounded adaptive learning, 21 specialists, 156 action types, opt-in gaze and 3D gesture paths, guarded neural-research workflows, and more truthful desktop execution.
Adaptive intelligence
Verified experience, temporal memory, strategy evolution, and optional JEPA-style prediction advise plans without widening authority.
21 specialists · 156 actions
Concrete provider coverage spans desktop, browser, development, integrations, and bounded research workflows.
Reliable multimodal control
Voice, gaze, 3D hand tracking, cursor mode, and gesture workflows can run together with temporal false-positive rejection.
Guarded neural research
Synthetic BrainFlow and recorded EEGBCI evidence is available without misrepresenting it as proven live brain control.
Truthful execution
Application resolution, browser targeting, approvals, cancellation, result reporting, and first-run setup are more deterministic.
Current main software validation
1,653 Python tests passed · 179 frontend tests passed · cross-platform visual and Rust checks passed · 0 high-severity npm vulnerabilities
Physical microphone, camera, gaze, gesture, audible TTS, and live-EEG accuracy remains device-dependent.
Measured software evidence · 13 August 2026
Benchmarks with boundaries.
The public evidence bundle records distributions, regression controls, raw data, reproduction commands, and known exclusions. These are software-path measurements on one Windows host—not universal device, provider, network, or human-accuracy claims.
Guarded local path
28.640 ms
Median across 100 ready-daemon CPU-status requests; 30.490 ms p95 and zero model calls.
Intent controls
59 / 59
A fixed deterministic routing regression set, not population-level language accuracy.
Shared-loop health
65 ticks
Concurrent scheduler heartbeats during a real one-second CPU sample; 16.296 ms maximum gap.
Learned risk
36k / 5.4k
Training and temporal-validation samples for a bounded disk/process predictor; deterministic policy remains authoritative.
What these numbers exclude
Model-provider and network latency, browser page loads, UI rendering, microphone capture, audible TTS, camera, gesture, gaze, live EEG, and human accuracy all require separate evidence.
Get Started
Download the compiled desktop app or build from source for development.
Windows
1.
Download Heliox-OS_0.11.1_x64-setup.exe or the .msi from the latest published release.
2.
Install and open Heliox OS.
3.
Enter your API key (Gemini, OpenAI, Claude, Meta) in the Settings tab.
macOS
1.
Download for Apple Silicon: Heliox-OS_0.11.1_aarch64.dmg
2.
Download for Intel: Heliox-OS_0.11.1_x64.dmg
3.
Mount, drag to Applications, and configure your API key in Settings.
Linux
1.
Choose your format: .AppImage, .deb, or .rpm
2.
Install and run. Requires Python 3.11+.
3.
Configure API key in Settings. First launch may need time for env initialization.
# Clone and install daemon
git clone https://github.com/VyomKulshrestha/Heliox-OS.git
cd Heliox-OS/daemon
pip install -e ".[all,dev]"
# Run the daemon
python -m pilot.server
# Run the frontend (separate terminal)
cd tauri-app/ui && npm ci && npm run dev
Requires Python 3.11+. The desktop app starts the local daemon automatically. Browser-only UI development requires a manually started daemon.
Try It
Natural language commands for anything on your system — from simple queries to complex multi-step automations.
"Show me my system info"
"Kill the process using the most CPU"
"Install Firefox"
"Set my volume to 50%"
"Create a Python project with tests"
"Go to Wikipedia and summarize AI"
"Monitor CPU above 80%"
"List all .py files on Desktop"
1
import heliox
2
from heliox.core import Agent
3
4
# Initialize with 21 specialist agents
5
agent = Agent(name="Heliox-Alpha", mode="Autonomous")
6
agent.spawn_agent(permissions=["sys_control", "vision"])
7
8
agent.execute_loop(prompt="Organize my workspace")
9
>> Status: Agent Active. 21 specialists online.
Capability guides
What Heliox can do—without the hype
Plain-language guides explain the real workflow, safety boundary, hardware requirements, and known limits for every major interaction mode.
01
Voice control
Continuous listening, speech, approvals, and device limits.
02Browser + apps
Target resolution, visible actions, and verified outcomes.
03Hands-free access
Multiple input paths with independent stop controls.
04Gesture + gaze
On-device signals, temporal checks, and calibration.
05Autonomous work
Durable jobs and bounded execution—not unlimited authority.
06Plugin marketplace
Reviewed submissions, integrity checks, and capabilities.
07 · RESEARCH BOUNDARYRecorded-EEG research
Synthetic and recorded signal validation, clearly separated from any live brain-control claim.
Creator Profile
About Me
Hi, I'm Vyom Kulshrestha, a pre-final year Computer Science student at VIT, passionate about building next-generation AI systems and intelligent developer experiences.
I'm building Heliox OS — a futuristic AI-powered operating system interface designed to combine autonomous agents, voice interaction, multimodal intelligence, and system-level automation into a seamless computing experience.
Whether you're a developer, open-source contributor, recruiter, or someone excited about the future of human-computer interaction, I'd love to connect.
Ready to supercharge your OS?
An open-source project built solo, growing with every contributor. Be part of what comes next.