VRHandsFrame Mods and User Extension Specification (1.0.1)

Introduction

You can use some external API by installing official mods on the Steam Workshop.
Please obtain your own authentication information such as API key.

For Image Recognition

VisionBaseMod: https://steamcommunity.com/sharedfiles/filedetails/?id=3393878913

  Default Supported APIs (1.0.1):
  -  Google Cloud Vision (DOCUMENT_TEXT_DETECTION)
  -  Microsoft Azure AI Vision (Read API)
  - *OpenAI Vision Sample (gpt4o-mini)

  * As an example of user extensions

For Translation

TranslationBaseMod: https://steamcommunity.com/sharedfiles/filedetails/?id=3393879278

  Default Supported APIs (1.0.1):
  -  DeepL API
  -  Microsoft Azure AI Translator
  - *Google App Script Sample (VRHandsFrame v1 compatible)
  - *LibreTranslate

  * As an example of user extensions

When you install the mod, an item will appear at the bottom of the sidebar on the settings screen.
Turn the slide bar ON to use Mods instead of built-in functions.
If the Mods item does not appear, please restart VRHandsFrame or Steam client.

Mods Item

About User Extensions

If you want to use an API that is not provided by the official mods, there are two ways to do so:

Add new json files that describe Extension Object. (recommended)
Edit official user_extension.json file

Extension Object is used to define API URL endpoint, request/response params, and other information for user extensions.
The user_extension.json file is a file that contains some example Extension Object.

File Location

When you install a mod from the Steam workshop, required user_extension.json and manifest.json file is generated in the following directory on application startup:

For Image Recognition

../SteamLibrary/steamapp/common/VRHandsFrame/Mods/VisionBaseMod/

For Translation

../SteamLibrary/steamapp/common/VRHandsFrame/Mods/TranslationBaseMod/

At this location, add your json files that describe Extension Object.
You can give any file name, but the value of the "name" key in the Extension Object must be unique by each mod.

** Don't Delete the original user_extension.json and manifest.json file! These files are required. **

Mods Usage Guidelines

Please comply with the VRHandsFrame Software License Agreement.
https://store.steampowered.com//eula/2324090_eula_0
Do not use in a way that steals other user's data.
Do not use for commercial advertising of any hosting services (such as selling API keys or subscriptions).
When using an user extension created by someone else, be careful to ensure that no suspicious URLs are set.
VRHandsFrame App developer (Eureka Works) will not store or view any user's data when using both mods of image recognition and translation.
Please pay attention to the terms of use and privacy policy of each API service regarding data handling.

Extension Object

name required	string Extension name It will be displayed on VRHandsFrame settings menu. The name must be unique by each mod. It must not overlap with any other Extension Object name.
url required	string Destination URL for API requests
required	Key (object) Authentication information that serves as the key for API requests. It can be pasted or deleted from the VRHandsFrame settings menu. When publishing your extension configuration, be careful not to include the key directly.
	object Optional HTTP headers for API requests
required	Array of APIRequestObject (object) non-empty Required parameters for API requests Requests are sent using the HTTP POST method. Note that this is an Array object with multiple types of parameter elements.
required	APIResponseObject (object) Expected parameters for API responses

Example

OpenAI Vision Sample(gpt4o-mini)

{"name": "OpenAI Vision Sample (gpt-4o-mini)",
"url": "https://api.openai.com/v1/chat/completions",
"key": {"type": "bearer",
"value": ""
},
"headers": {"Content-Type": "application/json"
},
"request": [{"type": "json",
"params": {"model": "gpt-4o-mini",
"messages": [{"role": "user",
"content": [{"type": "text",
"text": "What is in this image?"
},
{"type": "image_url",
"image_url": {"url": "data:image/png;base64,$image_data"
}
}
]
}
],
"max_tokens": 1024
}
}
],
"response": {"type": "json",
"jsonKey": "choices[0].message.content"
}
}

API Request Object

type required	string Enum: "json" "form" "query" Parameter type json : JSON format form : HTTP POST form format query : HTTP URL query parameter format The query parameter object can be combined with JSON or form data. Define them separately as array elements.
params required	object API request parameters Set the parameters according to the specifications of the API you want to use. The following strings are used as placeholders: $image_data : The PNG image data to be recognized (Base64 encoded)

Example

vision

{"type": "json",
"params": {"image": "$image_data"
}
}

API Response Object

type required	string Value: "json"
jsonKey required	string Json Key that stores the text information you want to receive It must be provided in a format that can be parsed by the JToken.SelectToken Method of the Json.NET library. https://www.newtonsoft.com/json/help/html/SelectToken.htm

Example

json

{"type": "json",
"jsonKey": "choices[0].message.content"
}

user_extension.json

description required	string Mod name
version required	string Mod version
required	Array of ExtensionObject (object) >= 0 items User extension definitions

Example

VisionBaseMod(1.0.1)

{"description": "Vision User Extension",
"version": "1.0.1",
"extensions": [{"name": "OpenAI Vision Sample (gpt-4o-mini)",
"url": "https://api.openai.com/v1/chat/completions",
"key": {"type": "bearer",
"value": ""
},
"headers": {"Content-Type": "application/json"
},
"request": [{"type": "json",
"params": {"model": "gpt-4o-mini",
"messages": [{"role": "user",
"content": [{"type": "text",
"text": "What is in this image?"
},
{"type": "image_url",
"image_url": {"url": "data:image/png;base64,$image_data"
}
}
]
}
],
"max_tokens": 1024
}
}
],
"response": {"type": "json",
"jsonKey": "choices[0].message.content"
}
}
]
}