Page 1 of 2 12 LastLast
Results 1 to 10 of 12

Thread: HumanMouse Development Diary

  1. #1
    Join Date
    May 2011
    Location
    https://runelite.plus/
    Posts
    78
    Mentioned
    2 Post(s)

    HumanMouse Bot - Development Diary

    HumanMouse Development Diary

    Hi guys, thought I'd share some details on a project I'm working on for personal use.
    I am essentially recreating my own version of RiD Genesis for fun, I thought you guys might find this project interesting.


    All of my updates can be found in the following post. I have moved the original thread to the change log as the first entry.

    Last edited by soxs; 11-19-2018 at 07:45 PM.

  2. Snoopy, Early, Nixter243, Drum, Jasam, iStokee liked this post
  3.  
  4. #2
    Join Date
    May 2011
    Location
    https://runelite.plus/
    Posts
    78
    Mentioned
    2 Post(s)
    Latest Log:
    19/11/2018
    This is a pretty good time to tell you guys that I've decided to go a pretty different direction with the project and so as such this will be a pretty large change log.
    I will try to explain everything as well as I can, as I know some of you actually find this stuff pretty interesting.

    Before I can explain what the changes are, I need to explain the problems that I was facing trying the initial ideas.
    • I wanted to use a neural network for the majority of the decision making in this bot, including mouse movements (and object recognition on the screen which I will get to next).
      The problem with neural networks is that while they are great at classification problems like these, they can quickly become slow and inefficient once you feed it too many inputs.
      In my implementation the inputs were a list of normalised previous x & y values (tuples), and the output was a prediction of the next tuple. The issue I faced is that when the net was given the same set of previous tuples it would give the same next tuple, generating the same path again.
    • I wanted to use a screen scraper to gather information on the screen. The further I got with implementing this, the more I realised that this was very complex in order to make it a reliable solution. Even if I got object recognition to work the way I wanted it to, I would need to implement OCR in order to read text on the screen. While this could have been a viable approach (such as RiD's implementation), it takes too long to program and is too convoluted for a project as trivial as mine.


    So now you understand the position I was in ~3 weeks ago, I had to come up with something different.

    Mouse movements:
    I decided to use a method I came up with derived from "path splicing". I essentially applied an exploded-view logic to the mouse paths.
    The applied algorithm splits the mouse paths into smaller paths at each point where the turn angle exceeds a set thresh-hold (I've been playing around between 5 and 15 degrees). Heres an image to help explain:
    • The first image is the original mouse path.
    • The second image is the mouse paths "break-points", the points where the angle exceeded the maximum.
    • The third image is the collected "exploded paths", the paths in-between the previously mentioned "break-points".

    Spoiler: 


    Using this data and some randomisation, my algorithm generates similar mouse paths in around a millisecond each.
    The more biometric data collected, the more variations of mouse path it can generate.
    Heres an example of 10 mouse paths which took 10ms to generate based from one original mouse path:
    (some further processing and smoothening will be done to fix the jaggedness)



    Object Recognition:
    I ultimately decided that instead of reading the screen, it would be faster and more accurate to use reflection to grab values out of the game-pack.
    Getting values out of the game-pack requires programming a hook updater to detect patterns and identify fields and methods. Although I do have my own hook updater for OSRS, they're tedious to manage so instead of reflecting the game-pack directly, I decided to reflect the popular, open-source, RuneLite client.
    Currently I use java-agents and reflection to run my application in the same virtual machine as RuneLite, allowing for shared resources between classes.
    I will probably change this to some form of resource bridging protocol between two separate JVM's in the future.

    Getting values from RuneLite: (such as current level, xp, position etc)


    So that's where I'm currently at. First I need to finish processing those generated mouse paths to make them smoother, then I need to create some form of scripting framework and start testing some basic tasks.

    -

    Older Logs:


    24/10/2018
    Spoiler: 
    Added some more processing to the paths to normalise them further. I don't think anything else is required for them so I can start tweaking the neural network stuff so that it can generate human-like mouse paths.
    I also added some UI stuff, serialisation (saving/loading biometric data) and some other stuff.

    Spoiler: 



    23/10/2018
    Spoiler: 

    I have started with simulating human biometrics for mouse movements. I am tracking the path, time taken to complete path and magnitude of the path. Each path is normalised as to always begin from point (0, 0) and always end along the positive side of the x-axis.
    Spoiler: 


    Next I need to train the neural network I have created with the collected mouse data, in order to simulate my mouse movements.

    I'll keep this updated
    Last edited by soxs; 11-19-2018 at 07:49 PM. Reason: update log

  5. #3
    BillyS is offline
    Veteran Member  
    $10 Credit    Premium: 1500->2000 hours used  Free Bots: 100->200 hours used  99 Fishing 99 Fletch 99 Herblore 99 Crafting 99 Cook 99 Miner 99 Smith 99 Mager 
    Join Date
    Oct 2012
    Posts
    33
    Mentioned
    0 Post(s)
    Cool, super interesting to read. This is the new generation of computer programming and you make it sound so simple. How much experience have you had in AI programming?
    Last edited by BillyS; 10-25-2018 at 01:21 AM.

  6. #4
    Join Date
    May 2011
    Location
    https://runelite.plus/
    Posts
    78
    Mentioned
    2 Post(s)
    Quote Originally Posted by BillyS View Post
    Cool, super interesting to read. This is the new generation of computer programming and you make it sound so simple. How much experience have you had in AI programming?
    Its a new generation of runescape botting, but these concepts have existed for a long time.
    At university I studied computer games programming and chose AI as my optional modules throughout.
    The vector operations (like rotation matrix) are actually really common and used in games regularly, and the AI such as neural networks have been around for decades.

  7. #5
    Join Date
    Dec 2014
    Location
    Florida
    Posts
    590
    Mentioned
    35 Post(s)
    Very neat, interested on the outcome when it comes to detection.

  8. #6
    Join Date
    May 2011
    Location
    https://runelite.plus/
    Posts
    78
    Mentioned
    2 Post(s)
    Quote Originally Posted by xcendrox View Post
    Very neat, interested on the outcome when it comes to detection.
    I'm pretty curious too, guess we'll just see how it performs when it's ready.

  9. #7
    Join Date
    May 2011
    Location
    https://runelite.plus/
    Posts
    78
    Mentioned
    2 Post(s)
    19/11/2018
    This is a pretty good time to tell you guys that I've decided to go a pretty different direction with the project and so as such this will be a pretty large change log.
    I will try to explain everything as well as I can, as I know some of you actually find this stuff pretty interesting.

    Before I can explain what the changes are, I need to explain the problems that I was facing trying the initial ideas.
    • I wanted to use a neural network for the majority of the decision making in this bot, including mouse movements (and object recognition on the screen which I will get to next).
      The problem with neural networks is that while they are great at classification problems like these, they can quickly become slow and inefficient once you feed it too many inputs.
      In my implementation the inputs were a list of normalised previous x & y values (tuples), and the output was a prediction of the next tuple. The issue I faced is that when the net was given the same set of previous tuples it would give the same next tuple, generating the same path again.
    • I wanted to use a screen scraper to gather information on the screen. The further I got with implementing this, the more I realised that this was very complex in order to make it a reliable solution. Even if I got object recognition to work the way I wanted it to, I would need to implement OCR in order to read text on the screen. While this could have been a viable approach (such as RiD's implementation), it takes too long to program and is too convoluted for a project as trivial as mine.


    So now you understand the position I was in ~3 weeks ago, I had to come up with something different.

    Mouse movements:
    I decided to use a method I came up with derived from "path splicing". I essentially applied an exploded-view logic to the mouse paths.
    The applied algorithm splits the mouse paths into smaller paths at each point where the turn angle exceeds a set thresh-hold (I've been playing around between 5 and 15 degrees). Heres an image to help explain:
    • The first image is the original mouse path.
    • The second image is the mouse paths "break-points", the points where the angle exceeded the maximum.
    • The third image is the collected "exploded paths", the paths in-between the previously mentioned "break-points".

    Spoiler: 


    Using this data and some randomisation, my algorithm generates similar mouse paths in around a millisecond each.
    The more biometric data collected, the more variations of mouse path it can generate.
    Heres an example of 10 mouse paths which took 10ms to generate based from one original mouse path:
    (some further processing and smoothening will be done to fix the jaggedness)



    Object Recognition:
    I ultimately decided that instead of reading the screen, it would be faster and more accurate to use reflection to grab values out of the game-pack.
    Getting values out of the game-pack requires programming a hook updater to detect patterns and identify fields and methods. Although I do have my own hook updater for OSRS, they're tedious to manage so instead of reflecting the game-pack directly, I decided to reflect the popular, open-source, RuneLite client.
    Currently I use java-agents and reflection to run my application in the same virtual machine as RuneLite, allowing for shared resources between classes.
    I will probably change this to some form of resource bridging protocol between two separate JVM's in the future.

    Getting values from RuneLite: (such as current level, xp, position etc)


    So that's where I'm currently at. First I need to finish processing those generated mouse paths to make them smoother, then I need to create some form of scripting framework and start testing some basic tasks.
    Last edited by soxs; 11-19-2018 at 07:50 PM.

  10. dztancem8, pppffft0, iStokee liked this post
  11.  
  12. #8
    Join Date
    May 2011
    Location
    https://runelite.plus/
    Posts
    78
    Mentioned
    2 Post(s)
    been a long time since i worked on this.

  13. #9
    Join Date
    Jan 2010
    Posts
    58
    Mentioned
    0 Post(s)
    Cool read , I’m sure RiD could use a partner

  14. soxs liked this post
  15.  
  16. #10
    Join Date
    Aug 2019
    Posts
    2
    Mentioned
    0 Post(s)
    Very interesting topic!

    Do you by any chance know where I could find any tutorial related to hooks & reflection bot scripting?
    I'm very new to the branch of botting but I would love to learn more about how to utilizing a game client's libraries and hooking in your own scripts.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •