Gtuner IV 1.07 PUBLIC RELEASE: Computer Vision

Official news and announcements from the ConsoleTuner team.

Gtuner IV 1.07 PUBLIC RELEASE: Computer Vision

Postby J2Kbr » Sun Feb 09, 2020 9:38 am

-- UPDATE (March 14th, 2020) --

Gtuner IV 1.07 Public Release

-- UPDATE (March 6th, 2020) --

Gtuner IV release candidate is now available for testing. In this 1.07 final testing release is now possible add an URL/link for donation when publishing or updating a work (thank you Mad for the idea). With that the author can receive donations from users as gratitude for the hard work and time spend. The donation button will be shown below the work information on the right side and will work on all Gtuner versions (current and previous), as well as at the ConsoleTuner.com Online Resource viewer.

Gtuner IV 1.07RC Changelog:
* Qt framework updated to 5.14.1
* Preferences: Added option to manually enable automatic scaling for high DPI monitors
* Session: The order of files opened in the GPC Script IDE are now correctly restored
* Memory Slot: Fixed issue with the configuration button not being displayed after script drop
* Gtuner IV: Added splash screen for early indication Gtuner IV is launching
* Computer Vision panel for view video capture, select source, run CV scripts, save frame as image
* Computer Vision: Input source configuration interface (device, stream, display, window, null)
* Computer Vision: integration with Python executable library and OpenCV-Python module
* Preferences: Added CV pane to select Python location and check if files needed were found
* Preferences (Test and Debug): Added option to automatic open the CV panel for GCV scripts
* File Explorer: Added icon, enabled listing and opening of GCV script files (*.py)
* Gtuner Menu and Toolbar: added functionalities to create and manage GCV script files (*.py)
* Code Editor: Added custom lexer, highlight, auto-completion for GCV script files (*.py)
* Computer Vision: Accept GPC script commands "GCMD:ComputerVision.[Stop|Start|Start:<gcvfile>]"
* GPC Compiler: New functions gcv_ready() and gcv_read() to receive feedback from GCV scripts
* Publishing: Accept GCV scripts and referred supporting files for CV algorithms (images, xml)
* Online Resources: Changed category 'Macro' to 'Misc' for list GMK macros and GCV scripts
* Gtuner Update: Launch Gupdate helper application detached from Gtuner IV process
* Computer Vision: Built-in gtuner Python module to read the input, output, LED and FFB states
* Computer Vision: Use OpenGL for the frame viewport (faster frame drawing, less CPU usage)
* Computer Vision: Fixed horizontal shifting in the export frame to image operation {GitHub#359}
* Header &lt;titanone.gph&gt;: Fixed error in the algorithm of stickize() function {GitHub#360}
* Output Panel: Print debug messages related with Bluetooth Module connection errors
* Code Editor: Auto select corresponding file type for 'Save As' and 'Save Copy As' {GitHub#358}
* Gtuner Mac: Workaround for OSX Catalina not reporting removal of device, causing Gtuner to crash
* Gtuner Mac: Computer Vision, with default kern.sysv.shmmax viewport is 720p, increase for 1080p
* Gtuner Mac: Several GUI layout fixes and application style changed to 'Fusion' for better look
* Publishing: Added extra field for the author provide URL for donations (optional) {GitHub#364}
* Code Editor: Fixed bug with 'Save As' and 'Save Copy As' for GCV script files {GitHub#363}
* GPC Compiler: Added MOUSE_ACPAN definition to mouse.gph header file (horizontal scroll)


-- UPDATE (February 25th, 2020) --

Gtuner 1.07B-20 for Windows and now for Mac OS X, download links:


-- UPDATE (February 19th, 2020) --

New revision of the Gtuner IV 1.7-BETA is now available, download link [removed, official release out.]

Changes on this revision (1.7.1.15)

Implemented built-in 'gtuner' Python module to read from a GCV Script the input, output, LED and FFB states of the devices connected to the Titan Two.

GCV code example:
Code: Select all
import cv2
import gtuner

class GCVWorker:
    def __init__(self, width, height):
        pass
   
    def __del__(self):
        pass
   
    def process(self, frame):
        # Read input states
        stick_1_x = gtuner.get_actual(gtuner.STICK_1_X)
        stick_1_y = gtuner.get_actual(gtuner.STICK_1_Y)
        button_5 = gtuner.get_actual(gtuner.BUTTON_5)
        button_8 = gtuner.get_actual(gtuner.BUTTON_8)
       
        # Read output states
        button_14 = gtuner.get_val(gtuner.BUTTON_14)
        button_15 = gtuner.get_val(gtuner.BUTTON_15)
        button_16 = gtuner.get_val(gtuner.BUTTON_16)
        button_17 = gtuner.get_val(gtuner.BUTTON_17)
       
        # Read LED and FFB states
        led_1 = gtuner.led_get(gtuner.LED_1)
        ffb_2 = gtuner.ffb_get(gtuner.FFB_2)
       
        # Show state values
        frame = cv2.putText(frame, '{0:+6.2f}'.format(stick_1_x), (60,  80), cv2.FONT_HERSHEY_SIMPLEX, 1, (255, 255, 255), 2, cv2.LINE_AA)
        frame = cv2.putText(frame, '{0:+6.2f}'.format(stick_1_y), (60, 120), cv2.FONT_HERSHEY_SIMPLEX, 1, (255, 255, 255), 2, cv2.LINE_AA)
        frame = cv2.putText(frame, '{0:+6.2f}'.format(button_5),  (60, 160), cv2.FONT_HERSHEY_SIMPLEX, 1, (255, 255, 255), 2, cv2.LINE_AA)
        frame = cv2.putText(frame, '{0:+6.2f}'.format(button_8),  (60, 200), cv2.FONT_HERSHEY_SIMPLEX, 1, (255, 255, 255), 2, cv2.LINE_AA)
        frame = cv2.putText(frame, '{0:+6.2f}'.format(button_14), (60, 240), cv2.FONT_HERSHEY_SIMPLEX, 1, (255, 255, 255), 2, cv2.LINE_AA)
        frame = cv2.putText(frame, '{0:+6.2f}'.format(button_15), (60, 280), cv2.FONT_HERSHEY_SIMPLEX, 1, (255, 255, 255), 2, cv2.LINE_AA)
        frame = cv2.putText(frame, '{0:+6.2f}'.format(button_16), (60, 320), cv2.FONT_HERSHEY_SIMPLEX, 1, (255, 255, 255), 2, cv2.LINE_AA)
        frame = cv2.putText(frame, '{0:+6.2f}'.format(button_17), (60, 360), cv2.FONT_HERSHEY_SIMPLEX, 1, (255, 255, 255), 2, cv2.LINE_AA)
        frame = cv2.putText(frame, '{0:+6.2f}'.format(led_1),     (60, 400), cv2.FONT_HERSHEY_SIMPLEX, 1, (255, 255, 255), 2, cv2.LINE_AA)
        frame = cv2.putText(frame, '{0:+6.2f}'.format(ffb_2),     (60, 440), cv2.FONT_HERSHEY_SIMPLEX, 1, (255, 255, 255), 2, cv2.LINE_AA)
        return (frame, None)


-- UPDATE (February 18th, 2020) --

New revision of the Gtuner IV 1.7-BETA is now available, download link: [removed, official release out.]

Changes on this revision (1.7.1.10)

Improved Computer Vision Toolbar:
GtunerCVB10A.png
GtunerCVB10A.png (5.48 KiB) Viewed 33765 times


Support for wide range of video inputs:


- Video Capture Devices (such as Capture Cards, Webcams)
- Network Video Streams (such as RTSP/H264 video stream)
- Display Capture (with support for multi-monitors)
- Window Capture (select the application to be captured)
- Null Video Input (run GCV scripts that does not requires video input)

Note: Display Capture and Window Capture requires PyWin32 module to be installed.
GtunerCVB10C.png
GtunerCVB10C.png (23.48 KiB) Viewed 33765 times


Fully customizable video input configuration:
GtunerCVB10B.png
GtunerCVB10B.png (14.29 KiB) Viewed 33765 times

For video capture and network streaming is now possible to manually choose the media backend. In the first beta release the backend in use was MS Media Foundation. If your capture card didn't work at that time, try again using MS DirectShow, which is an old backend, but has support for a greater variety of video capture devices.

Easy setup: Click on "Detect Devices, "Detect Displays" or "Detect Windows" button, select one option and hit OK to fill all relevant configurations automatically.

Advanced Configurations:

For video capture and network streaming the key-value pair can be any from the OpenCV API, as defined here:
https://docs.opencv.org/4.2.0/d4/d15/gr ... f7c4f5704d
and here:
https://docs.opencv.org/4.2.0/dc/dfc/gr ... thers.html

For Display Capture the valid keys are:
Code: Select all
CAP_CROP_TOP
CAP_CROP_BOTTOM
CAP_CROP_LEFT
CAP_CROP_RIGHT
In addition the the above, Window Capture also accepts:
Code: Select all
CAP_OFFSET_X
CAP_OFFSET_Y

enjoy! :smile0517:

-- ORIGINAL POST --
Gtuner IV 1.07 is now available for beta testing (Windows only).

This release adds a scriptable Computer Vision Interface to Gtuner IV, allowing the Titan Two users download, create and share computer vision algorithms (GCV scripts) that analyzes video streaming from games or others video sources such as webcams. The computer vision script can communicate with the Titan Two device to provide information for the running GPC Script.

The possibilities for this new feature are endless, to illustrate:

  • - Automated detection of the game aiming mechanics for creation of optimized mouse to stick translators (yx ratio, dd factor, conversion curve, deadzone, etc).
    -
  • - Detect and inform to the Titan Two script the equipped gear and game status (weapon model, weapon attachments, fire mode, shots fired, etc).
    -
  • - Analyzing of the game minimap to alert the player about any important event (so the player can fully concentrate on the actual game and stop look to the minimap every couple seconds).
    -
  • - Detect and register weapon recoil/spray pattern, so the collected data can be used for creation of anti-recoil scripts.
    -
  • - Pattern detection and triggering of automated actions, such as:

    • * auto switch to secondary weapon if detect the primary weapon was fully unloaded;
      -
    • * auto block if an incoming attack is detected;
      -
    • * auto fire if detected an enemy is aligned with the aiming reticle.
      -
    • * and so on...

A example of GCV script is available on Gtuner's Online Resource, at "Misc Files" category.

GCV scripts are programmed in Python language and defined as a Python Class with name 'GCVWorker'. The class has a constructor '__init__' that is executed when the script loads, here is where the programmer can allocate, load and initialize any resource needed for the computer vision algorithm, as well as be informed of what will be the resolution of captured video frames. The destructor '__del__' is executed when the script is about to unload and should be used to free any allocated resource. The main method is named 'process' and is called whenever a new frame is captured. The process method can return up to two values: the frame to be displayed on Gtuner IV and/or an array of up to 255 bytes to be send to the Titan Two script. If the returning value for the frame is None, Gtuner IV will display the original captured frame. If the byte array is None, no data will be send to the Titan Two on that frame iteration.

Minimal GCV script example:
Code: Select all
import os
import cv2

class GCVWorker:
    def __init__(self, width, height):
        os.chdir(os.path.dirname(__file__))
        self.gcvdata = bytearray([0x00])
   
    def __del__(self):
        del self.gcvdata
   
    def process(self, frame):
        return (frame, None)

The Titan Two script can read the data from the computer vision algorithm by means of gcv_ready() and gcv_read(), as described here:

https://www.consoletuner.com/wiki/index ... :gcv_ready
https://www.consoletuner.com/wiki/index ... 2:gcv_read

Download the full beta release package from the following link: [removed, official release out.]

!!ATTENTION!!
Don't update your current Gtuner installation, otherwise you will get 'zlibwapi.dll was not found' error.

--

Any feedback on beta release will be very appreciated.

Thank you.

ORIGINAL POST wrote:Hi All, This post is to give you guys an update on what I am working these past weeks.

First, a background to set the context. By design, one of primaries requirements for the Titan Two is the "native" support for keyboard and mouse gameplay, meaning the mouse and keyboard could be directly plugged to the device without rely on computer connection. This feature is long supported by the Titan Two.

However, find the best mouse conversion parameters is not an easy task and normally it is a very time consuming process when doing manually. Since early stages of the Titan Two development the plan is implement some sort of computer vision tool to automate this process and find the most optimal mouse settings for each game.

November last year I finally started work on this tool, as the development progressed I thought: why not have this tool integrated on Gtuner VI, so all Titan Two users could make use of it to find the best mouse settings? sounds good, lets do it.

Few weeks later, I had another thought: this tool can also be used for others purposes, such as analyze the game frames to provide valuable information for the Titan Two script (e.g. equipped weapon, fire mode, etc), or even more advanced usages like have as video input source a webcam and use computer vision algorithms to aid disabled person to play their favorite games (e.g. activate the jump action with movement of a hand).

The ideas didn't stop come, the possibilities are endless. However, having the computer vision algorithms hard coded on Gtuner IV would impose an insurmountable barrier for these possibilities, as it would direct depend on me to implement each.

Okay, let have the computer vision algorithms as Gtuner IV plugins, so anyone interested could make their own. Sounds good, but it would require installation of compilers, computer vision libraries and advanced knowledge of computer programming. Not an accessible approach.

So what about have scripts to implement the computer vision algorithms? This would definitively be a more accessible option. Moreover if fully integrated on Gtuner IV.

So I did and here is the result:
CV1.png
CV1.png (1.69 MiB) Viewed 35441 times
CV2.png
CV2.png (143.58 KiB) Viewed 35441 times

The computer vision scripts are based on the OpenCV library and programmed utilizing Python script language.

It is required to have Python 3 x64 installed, as well as the module 'OpenCV-Python':
CV3.png
CV3.png (14.87 KiB) Viewed 35441 times

At some point next week Gtuner VI 1.07 will be made available as beta release (in the beta stage will be Windows only).
ConsoleTuner Support Team
User avatar
J2Kbr
General of the Army
General of the Army
 
Posts: 20323
Joined: Tue Mar 18, 2014 1:39 pm

Re: Computer Vision with Gtuner IV and Titan Two

Postby UK_Wildcats » Sun Feb 09, 2020 2:45 pm

Wow. That is a lot of work and great ideas. This is awesome.
User avatar
UK_Wildcats
Brigadier General
Brigadier General
 
Posts: 2243
Joined: Thu Jan 08, 2015 6:53 pm

Re: Computer Vision with Gtuner IV and Titan Two

Postby jelly44 » Mon Feb 10, 2020 8:53 am

J2Kbr wrote:However, find the best mouse conversion parameters is not an easy task and normally it is a very time consuming process when doing manually. Since early stages of the Titan Two development the plan is implement some sort of computer vision tool to automate this process and find the most optimal mouse settings for each game.

November last year I finally started work on this tool, as the development progressed I thought: why not have this tool integrated on Gtuner VI, so all Titan Two users could make use of it to find the best mouse settings? sounds good, lets do it.


This is great news, J2KBr!

I hope it does not require extensive coding knowledge to be able to work on mouse conversion parameters.

Awesome work, needless to say, on all the other features whereby experienced / advanced users will benefit greatly!
- My recommendation for learning C programming: 'C Programming Absolute Beginner's Guide'
- Create your own scripts with GPC2 Scripting
User avatar
jelly44
Sergeant Major of the Army
Sergeant Major of the Army
 
Posts: 220
Joined: Tue Feb 05, 2019 3:49 pm

Re: Computer Vision with Gtuner IV and Titan Two

Postby bonefisher » Mon Feb 10, 2020 9:57 am

:smile0203:
bonefisher
Lieutenant General
Lieutenant General
 
Posts: 5413
Joined: Thu Jan 29, 2015 10:49 am

Re: Computer Vision with Gtuner IV and Titan Two

Postby Prototype » Tue Feb 11, 2020 1:07 am

Wow, that is impressive! Great work, the possibilities are endless it's crazy!
Console tuner since my 1st controller.
Scripting, a game in the game.
Believe or dare, It's Titanic! :smile0517:
User avatar
Prototype
Major General
Major General
 
Posts: 3250
Joined: Sun Dec 16, 2012 1:43 pm

Re: Computer Vision with Gtuner IV and Titan Two

Postby antithesis » Tue Feb 11, 2020 6:29 am

It's great to see this coming together J2K :)

2020 is shaping to be a big year for Titan Two.
Official Australian retailer for Titan One, Titan Two and XIM APEX at Mod Squad
User avatar
antithesis
Colonel
Colonel
 
Posts: 1912
Joined: Sat May 28, 2016 10:45 pm

Re: Computer Vision with Gtuner IV and Titan Two

Postby dognose » Wed Feb 12, 2020 6:09 am

I'm excited about this new tech. Keep us updated!
User avatar
dognose
Master Sergeant
Master Sergeant
 
Posts: 30
Joined: Sun Nov 11, 2018 8:12 am
Location: South Los Angeles, CA

Re: Computer Vision with Gtuner IV and Titan Two

Postby hammer » Wed Feb 12, 2020 9:18 am

Cant wait for this !
User avatar
hammer
Sergeant Major
Sergeant Major
 
Posts: 104
Joined: Sun Sep 30, 2018 1:36 pm
Location: Germany

Re: Gtuner IV 1.07 BETA (Scriptable Computer Vision Interfac

Postby J2Kbr » Wed Feb 12, 2020 11:21 am

The beta release is now available, more details on the first post:

viewtopic.php?f=2&t=14717&p=95183#p95183
ConsoleTuner Support Team
User avatar
J2Kbr
General of the Army
General of the Army
 
Posts: 20323
Joined: Tue Mar 18, 2014 1:39 pm

Re: Gtuner IV 1.07 BETA (Scriptable Computer Vision Interfac

Postby Mad » Wed Feb 12, 2020 11:41 am

J2Kbr wrote:The beta release is now available

Got this error after updating Gtuner IV:
gtuner.png
gtuner.png (11.23 KiB) Viewed 34775 times
ConsoleTuner Support Team || ConsoleTuner Discord || InputSense Discord (2K / FPS)
Mad
Major General
Major General
 
Posts: 4532
Joined: Wed May 22, 2019 5:39 am

Next

Return to News and Announcements

Who is online

Users browsing this forum: mizamori and 56 guests