Open-source software for capturing Japanese text from screenshots and translating it using GPT-3.5.
The program requires you to have an account with OpenAI and an API key to use the models.
💡 How it works
1) Take a screenshot
Set up a keyboard shortcut to capture the area of the screen you want. You can use it to capture text from manga, video games, or any other source. We will be adding support for multiple monitors soon.
2) The system extracts the Japanese characters
We use Manga-OCR to analyze the capture and extract the Japanese text. We will be adding other character extractors such as EasyOCR soon to improve the system output and add language support.
3) The system uses GPT to translate the text. There are several options to control GPT
The text is sent to the OpenAI GPT-3.5 model to be translated. This step requires you to create an OpenAI account to obtain an API key. We will be adding models as they are released. We also plan to add other sources of translation, such as DeepL and Yandex.
🌐 Download and Use
The project is open-source and can be found on Github.
Here you will find more technical details about the system and instructions for running the source code as well as compiling your own executable. Collaborations to improve the system are welcome.
We also have an installable version that you can download from here:
After installing the program, the Manga-OCR model, which has a size of 450Mb, will be automatically downloaded.
❤️ From a developer, for the community
This project was born as a way to revive my previous project, “K-AVT”, which I had to abandon due to technical barriers. This new project has been built entirely from scratch to be more user-friendly, adaptable, lightweight, and performant. My goal is to make this program, VGT, realize the original idea of K-AVT: real-time translation of Japanese video games.
My initial motivation was to have a way to play Japanese titles that I doubted would ever be translated.
On the other hand, the new OpenAI models we have seen this year give us a real opportunity to experiment with translators based on large language models. My idea is to experiment with different prompts and mechanisms to obtain the best possible results. I also want to use different OCRs and translators in parallel and use the GPT models to combine the results.
Best regards to everyone, K-RT
📚 Example of translations by VGT
TODO