HumanMouse Development Diary
Hi guys, thought I'd share some details on a project I'm working on for personal use.
I am essentially recreating my own version of RiD Genesis for fun, I thought you guys might find this project interesting.
All of my updates can be found in the following post. I have moved the original thread to the change log as the first entry.
Last edited by soxs; 11-19-2018 at 06:45 PM.
Latest Log:
19/11/2018
This is a pretty good time to tell you guys that I've decided to go a pretty different direction with the project and so as such this will be a pretty large change log.
I will try to explain everything as well as I can, as I know some of you actually find this stuff pretty interesting.
Before I can explain what the changes are, I need to explain the problems that I was facing trying the initial ideas.
- I wanted to use a neural network for the majority of the decision making in this bot, including mouse movements (and object recognition on the screen which I will get to next).
The problem with neural networks is that while they are great at classification problems like these, they can quickly become slow and inefficient once you feed it too many inputs.
In my implementation the inputs were a list of normalised previous x & y values (tuples), and the output was a prediction of the next tuple. The issue I faced is that when the net was given the same set of previous tuples it would give the same next tuple, generating the same path again.- I wanted to use a screen scraper to gather information on the screen. The further I got with implementing this, the more I realised that this was very complex in order to make it a reliable solution. Even if I got object recognition to work the way I wanted it to, I would need to implement OCR in order to read text on the screen. While this could have been a viable approach (such as RiD's implementation), it takes too long to program and is too convoluted for a project as trivial as mine.
So now you understand the position I was in ~3 weeks ago, I had to come up with something different.
Mouse movements:
I decided to use a method I came up with derived from "path splicing". I essentially applied an exploded-view logic to the mouse paths.
The applied algorithm splits the mouse paths into smaller paths at each point where the turn angle exceeds a set thresh-hold (I've been playing around between 5 and 15 degrees). Heres an image to help explain:
- The first image is the original mouse path.
- The second image is the mouse paths "break-points", the points where the angle exceeded the maximum.
- The third image is the collected "exploded paths", the paths in-between the previously mentioned "break-points".
Spoiler:
Using this data and some randomisation, my algorithm generates similar mouse paths in around a millisecond each.
The more biometric data collected, the more variations of mouse path it can generate.
Heres an example of 10 mouse paths which took 10ms to generate based from one original mouse path:
(some further processing and smoothening will be done to fix the jaggedness)
Object Recognition:
I ultimately decided that instead of reading the screen, it would be faster and more accurate to use reflection to grab values out of the game-pack.
Getting values out of the game-pack requires programming a hook updater to detect patterns and identify fields and methods. Although I do have my own hook updater for OSRS, they're tedious to manage so instead of reflecting the game-pack directly, I decided to reflect the popular, open-source, RuneLite client.
Currently I use java-agents and reflection to run my application in the same virtual machine as RuneLite, allowing for shared resources between classes.
I will probably change this to some form of resource bridging protocol between two separate JVM's in the future.
Getting values from RuneLite: (such as current level, xp, position etc)
So that's where I'm currently at. First I need to finish processing those generated mouse paths to make them smoother, then I need to create some form of scripting framework and start testing some basic tasks.
-
Older Logs:
24/10/2018
Spoiler:
23/10/2018
Spoiler:
Last edited by soxs; 11-19-2018 at 06:49 PM. Reason: update log
| Veteran Member |  |
Cool, super interesting to read. This is the new generation of computer programming and you make it sound so simple. How much experience have you had in AI programming?
Last edited by BillyS; 10-25-2018 at 12:21 AM.
Its a new generation of runescape botting, but these concepts have existed for a long time.Originally Posted by BillyS
At university I studied computer games programming and chose AI as my optional modules throughout.
The vector operations (like rotation matrix) are actually really common and used in games regularly, and the AI such as neural networks have been around for decades.
Very neat, interested on the outcome when it comes to detection.
I'm pretty curious too, guess we'll just see how it performs when it's ready.
19/11/2018
This is a pretty good time to tell you guys that I've decided to go a pretty different direction with the project and so as such this will be a pretty large change log.
I will try to explain everything as well as I can, as I know some of you actually find this stuff pretty interesting.
Before I can explain what the changes are, I need to explain the problems that I was facing trying the initial ideas.
- I wanted to use a neural network for the majority of the decision making in this bot, including mouse movements (and object recognition on the screen which I will get to next).
The problem with neural networks is that while they are great at classification problems like these, they can quickly become slow and inefficient once you feed it too many inputs.
In my implementation the inputs were a list of normalised previous x & y values (tuples), and the output was a prediction of the next tuple. The issue I faced is that when the net was given the same set of previous tuples it would give the same next tuple, generating the same path again.- I wanted to use a screen scraper to gather information on the screen. The further I got with implementing this, the more I realised that this was very complex in order to make it a reliable solution. Even if I got object recognition to work the way I wanted it to, I would need to implement OCR in order to read text on the screen. While this could have been a viable approach (such as RiD's implementation), it takes too long to program and is too convoluted for a project as trivial as mine.
So now you understand the position I was in ~3 weeks ago, I had to come up with something different.
Mouse movements:
I decided to use a method I came up with derived from "path splicing". I essentially applied an exploded-view logic to the mouse paths.
The applied algorithm splits the mouse paths into smaller paths at each point where the turn angle exceeds a set thresh-hold (I've been playing around between 5 and 15 degrees). Heres an image to help explain:
- The first image is the original mouse path.
- The second image is the mouse paths "break-points", the points where the angle exceeded the maximum.
- The third image is the collected "exploded paths", the paths in-between the previously mentioned "break-points".
Spoiler:
Using this data and some randomisation, my algorithm generates similar mouse paths in around a millisecond each.
The more biometric data collected, the more variations of mouse path it can generate.
Heres an example of 10 mouse paths which took 10ms to generate based from one original mouse path:
(some further processing and smoothening will be done to fix the jaggedness)
Object Recognition:
I ultimately decided that instead of reading the screen, it would be faster and more accurate to use reflection to grab values out of the game-pack.
Getting values out of the game-pack requires programming a hook updater to detect patterns and identify fields and methods. Although I do have my own hook updater for OSRS, they're tedious to manage so instead of reflecting the game-pack directly, I decided to reflect the popular, open-source, RuneLite client.
Currently I use java-agents and reflection to run my application in the same virtual machine as RuneLite, allowing for shared resources between classes.
I will probably change this to some form of resource bridging protocol between two separate JVM's in the future.
Getting values from RuneLite: (such as current level, xp, position etc)
So that's where I'm currently at. First I need to finish processing those generated mouse paths to make them smoother, then I need to create some form of scripting framework and start testing some basic tasks.
Last edited by soxs; 11-19-2018 at 06:50 PM.
been a long time since i worked on this.
| RiD Member |  |
Cool read , I’m sure RiD could use a partner
soxs liked this post
Posting Permissions
Reply With Quote
iStokee
xcendrox
