Very often, when working on a computer, you have to perform a number of similar actions with a mouse, keyboard, etc. All this is exhausting and smart programmers (and they are always smart!), to make the user’s work easier, they wrote programs called autoclickers. Let's look at how you can avoid routine work using the example of the wonderful eMouse program.
eMouse
We go to the site and its with. Installation is not difficult. After installation, we launch it and a window appears. 
On the left is the “Rec” (record) button - after clicking, your actions with the mouse and keyboard will be recorded. “Play” will play (repeat the recorded actions). And how many times can be set in the “x” field (in the picture it is set 1 time), and put a tick in the “loop Playback” field
At the bottom left are the Record: Keys and Mouse checkboxes, i.e. if these checkboxes are checked, then mouse and keyboard actions will be recorded (and if they are not checked, then accordingly).
You can view (and change) the settings. Click on the “Edit” link and select “Settings” in the context menu
Here are the settings.
Hotkeys installed. To start recording actions - F5, to pause - F11, to play - F12.
Recorded actions (script) can be saved, i.e. you can write a bunch of different sequences and call them as needed. 
Open “File” and select “Save Script as...” and save it where we need it.
AutoClicker (the most primitive)
If you need to do a huge amount of clicking with the right or left mouse button, then this program is for you. It's great to use VKontakte in PROFIT games! Super Clicker or Clicker. 
The program is paid, but its “free” version can be downloaded from. Let's launch it. The program interface is quite simple. 
How to work with the left mouse button is written on the left side, and how to work with the right mouse button - on the right (we'll deal with the sliders later). We use hotkeys.
We launch the program, i.e. Press ALT+1 on the keyboard. This does not mean that we press the ALT key on the keyboard, then the “+” key, and then the key with the number “1!. No, we just press the “ALT” and “1” keys at the same time.
We go to VKontakte, look for a game where you need to make a bunch of clicks, move the mouse and press ALT+1. And until we press ALT+2, the clicks will continue.
Ghost Mouse can be downloaded from . Installation is not difficult. Let's launch. 
To record actions, press the red button. Or press F9. To play the actions, press “Ctrl+Q“.
Move Mouse is used to simulate the user's presence at the computer by moving the mouse cursor and pressing its buttons. In addition, you can use it to launch some programs, execute commands or PowerShell scripts. The utility can be activated manually or automatically, in the absence of user activity and according to a schedule, the same applies to stopping specified operations.
For each of the operations in the program's arsenal, additional options are available that allow you to set the distance and direction of mouse movement, select a button whose click will be simulated, specify arguments for a command, or hide the console window. All actions can be repeated (with a custom pause) or performed once.
In addition to auto-stop and auto-start, which we have already mentioned, in the Move Mouse behavior settings there is an option to automatically change the sound volume, hide the program window from the desktop, buttons and icons from the taskbar, thumbnails from the ALt+Tab dialog box and overwrite the title. The latest options will help hide traces of its presence and use on the computer, if someone needs it.
You can set up a schedule for activating actions using either a simple scheduler that allows you to select the day of the week and time to start or stop the program, or a more advanced one that supports entering Cron expressions. According to the schedule, Move Mouse can suspend its activities; the duration of the “blackout” also changes.
The program is distributed free of charge; in the current version, only one interface language is available - English. You can use Move Mouse on computers and tablets with Windows 10. Apparently, after publishing the program in the Windows Store, the developer refused to distribute it through other sources.
Install from Microsoft Store
In this series of articles I am going to show that, firstly: “the devil is not as terrible as he is painted,” and secondly: later I am going to answer the question: “why is this necessary?”
Now let's start with something simple. From establishing the relationship between the game and the player program (bot). The well-known game Zuma is taken as a guinea pig.
Any interaction consists of two processes: sending data to “them” and receiving data from “them”. In Zuma, all controls are done with the mouse, and the game provides feedback using images. Accordingly, first of all, you need to learn how to programmatically emulate the behavior of the mouse and obtain an image from the game.
The main goal of this article: to get a program that independently enters the game process over and over again, does something there, and when the game over-e starts all over again. Further, this framework will develop in the direction that the bot will last further and longer in the game until game over.
Solvable auxiliary subtasks: mouse emulation, mouse redirection to a virtual machine, image capture.
Retreat
When developing the code for this series of articles, the approach used is to get results as quickly as possible with a minimum of effort. This approach allows you to maintain motivation at a high level, and does not allow you to give up when faced with the insurmountability of the task. Because of this:- firstly, many insignificant (from the point of view of the current result) points will be quickly skimmed over, leaving “crutches and supports” in the code. And only in the next iterations will these points be analyzed separately, and the “crutches” will be replaced with full-fledged code.
- secondly, the code style is more “hacker” than classic C#. The code will contain a lot of lambdas, anonymous data, tricks, authorial discretion and a complete lack of comments.
Mouse emulation
Windows supports 2 standard methods of mouse emulation using 4 different WinApi functions.First way: sending the program its own window messages (WM_MOUSEMOVE, WM_LBUTTONDOWN, etc.) using the SendMessage or PostMessage functions.
For DirectX games (as in our case), this method is not suitable, because such programs for polling the mouse use DirectInput, which polls the mouse directly, ignoring Windows messages.
Second way: Direct emulation of mouse behavior using the mouse_event or SendInput functions. This method is suitable for any programs, including full-screen DirectX games. The mouse_event function is simpler, but it is considered obsolete; SendInput is more modern, but more cumbersome. Let's focus on mouse_event.
WinApi functions from C# are called using PInvoke technology. PInvoke descriptions for most common WinApi functions can be found on the PInvoke.net website. The mouse_event function is no exception.
public static extern void mouse_event(uint dwFlags, int dx, int dy, uint dwData, UIntPtr dwExtraInfo);
Mouse coordinates
The mouse_event function has a special feature: the mouse coordinates are specified in mickey, not in pixels. Conversion of mickey to pixels (and vice versa) depends on the resolution of the main monitor used. (0,0) corresponds to the upper left corner of the monitor, and (65535, 65535) to the lower right, which gives formulas for converting mickey to pixels and vice versa: mickey_point = pixel_point * (65536, 65536) / screen_size and pixel_point = mickey_point * screen_size / ( 65536, 65536) .Basic Operations
Summarizing all of the above, we get the following operations for controlling the mouse.Moving the mouse cursor to point (x,y):
mouse_event(MouseEventFlags.MOVE | MouseEventFlags.ABSOLUTE, x * 65536 / screen_width, y * 65536 / screen_height);
Left click:
mouse_event((MouseEventFlags.LEFTDOWN), 0, 0); System.Threading.Thread.Sleep(100); mouse_event((MouseEventFlags.LEFTUP), 0, 0);
Right click:
mouse_event((MouseEventFlags.RIGHTDOWN), 0, 0); System.Threading.Thread.Sleep(100); mouse_event((MouseEventFlags.RIGHTUP), 0, 0);
Problem: input exclusivity
There is a serious inconvenience when emulating a mouse through the mouse_event function: mouse_event simulates a mouse for the entire OS at once, and not for a separate application. It follows from this that while the bot is running and playing, other work on the computer is impossible: debugging the bot, actively viewing the bot’s state, reading the Internet, etc. But there is a way out: a virtual machine!Transferring a game to a virtual machine
Transferring the game to a virtual machine solves the following problems:- firstly, it simplifies interaction with games that do not support windowed mode and only work in full screen mode,
- secondly, mouse input is replaced only on the virtual machine, but on the main machine it continues to work in normal mode, allowing the computer user to go about his business.
The bot, unlike the game itself, is more convenient to run on the main machine. This allows you to restart the bot directly from Visual Studio, debug it there, have a place to display the internal state of the bot, etc.
Deploying a virtual machine (in this case, Oracle VirtualBox was used), installing a guest OS and transferring the game are done in a standard way, with the exception of one point: the bot needs the ability to establish communication over the network between the host OS and the guest OS. This is done in many ways. One way is to forward a specific port from the guest OS to the host OS using VirtualBox. Another way is to configure the Bridged Adapter mode, then the virtual machine will look like a regular computer for the entire network, and the guest OS will receive its IP address via dhcp from the router. Access from the host OS to the guest OS will take place at this address. (the author, in this case, used the option with a bridged adapter)
Proxy
To control the mouse on the guest OS, we will write a proxy, which is a simple console TCP server. Its full code is small and is presented below the cut. To simplify the code and reduce dependencies, the proxy is written on a bare socket without using remoting, wcf, etc. Proxy code
using System; using System.Collections.Generic; using System.Linq; using System.Net.Sockets; using System.Runtime.InteropServices; using System.Text; namespace InputProxy ( class Program ( static void Main(string args) ( var socket = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp); socket.Bind(new System.Net.IPEndPoint(System.Net.IPAddress. Any, 7001)); socket.Listen(10); for (; ;) ( var client = socket.Accept(); Console.WriteLine("connected.."); var thread = new System.Threading.Thread(( ) => ( try ( var clientReader = new System.IO.BinaryReader(new NetworkStream(client)); for (; ;) ( if (client.Poll(1, SelectMode.SelectRead) && client.Available == 0) ( Console.WriteLine("disconnected.."); break; ) if (client.Available > 0) ( var msgSize = clientReader.ReadInt32(); var message = clientReader.ReadBytes(msgSize); var messageReader = new System.IO. BinaryReader(new System.IO.MemoryStream(message)); var msgKind = messageReader.ReadInt32(); Console.WriteLine("message: kind:(0), len:(1)", msgKind, message.Length); switch (msgKind) ( case 0: ( var flags = messageReader.ReadUInt32(); var x = messageReader.ReadInt32(); var y = messageReader.ReadInt32(); var data = messageReader.ReadUInt32(); mouse_event(flags, x, y, data, UIntPtr.Zero); ) break; ) ) else System.Threading.Thread.Sleep(10); ) ) catch (Exception exc) ( Console.WriteLine(exc); ) )) ( IsBackground = true ); thread.Start(); ) ) public static extern void mouse_event(uint dwFlags, int dx, int dy, uint dwData, UIntPtr dwExtraInfo); ) )
For the proxy to work, you just need to copy it to a virtual machine and run it. The proxy waits for messages on port 7001 and displays a log of its operation on the console. To shut down the proxy, just close the console window.
Client
Connecting to a proxy is even simpler than the code for the proxy itself.var client = new System.Net.Sockets.TcpClient(vm_host, 7001); var clientStream = client.GetStream(); var clientWriter = new System.IO.BinaryWriter(clientStream); Action
Image interception
The easiest way to capture an image is directly from the screen. In .net there is a ready-made function for this, Graphics.CopyFromScreen. Let's look at this method in more detail.Firstly, you want to get a Bitmap as an output, not a Graphics - this can be solved using an auxiliary function:
public static Bitmap GetScreenImage(Rectangle rect) ( var bmp = new Bitmap(rect.Width, rect.Height, PixelFormat.Format32bppArgb); using (Graphics graphics = Graphics.FromImage(bmp)) ( graphics.CopyFromScreen(rect.Left, rect .Top, 0, 0, rect.Size, CopyPixelOperation.SourceCopy); ) return bmp; )
Secondly, you need to know which part of the screen you need to capture. You can, of course, always capture the same part of the screen and place the game with your hands in this part of the screen, but this is not convenient or practical. Moreover, automation of this process is done with minimal effort. WinApi and PInvoke will help us with this again, and more specifically two functions: FindWindow and GetWindowRect. FindWindow allows you to get the handle of the window using the window title, and GetWindowRect using the handle returns the position and size of the window on the screen.
There is a pinvoke description of both functions on the pinvoke.net website: FindWindow and GetWindowRect.
public static extern IntPtr FindWindow(string lpClassName, string lpWindowName); public static extern bool GetWindowRect(IntPtr hwnd, out RECT lpRect); public struct RECT ( public int Left; public int Top; public int Right; public int Bottom; )
And the code for capturing an image of a virtual machine window is as follows:
var vm_left = 8; var vm_right = 8; var vm_top = 50; var vm_bottom = 30; var vm_title = "Windows81 - Oracle VM VirtualBox"; var handle = FindWindow(null, vm_title); if (handle == IntPtr.Zero) throw new Exception("Окно не найдено"); RECT rect; GetWindowRect(handle, out rect); var gameScreenRect = new System.Drawing.Rectangle(rect.Left + vm_left, rect.Top + vm_top, rect.Right - rect.Left - vm_right - vm_left, rect.Bottom - rect.Top - vm_bottom - vm_top); var gameBmp = GetScreenImage(gameScreenRect); !}
Weakness
A significant drawback of this approach is that the captured window, firstly: must be entirely located on the screen, and secondly: must be located on top of all other windows. This inconvenience is leveled out with the help of two (or more) monitors :), then the virtual machine window is located on an auxiliary monitor, without disturbing anyone, remaining on top of the other windows. Also, this problem is completely solved using the previously discussed method: transferring the function (screen capture) inside the virtual machine. To do this, just add the appropriate function to InputProxy.Looping the gameplay
Finally, we proceed directly to solving the task set for today: looping the game process - all the necessary subtasks have been solved. The gameplay in Zuma revolves around three windows: main, mission and action. The main window contains the main menu, allowing you to select the type of game, the mission window prompts you to select a mission, and the action window is where the gameplay itself takes place.The bot determines the current window in the simplest way: by the color value at several key points. Points are selected manually using the “close gaze” method.
var screenChecks = new ( new ( Name = "main", Points = new ( new CheckPoint(200, 190, 0xff554a22), new CheckPoint(65, 400, 0xfff44c41) ) ), new ( Name = "mission", Points = new ( new CheckPoint(200, 190, 0xffb5d0c7), new CheckPoint(65, 400, 0xffad7630) ) ), new ( Name = "action", Points = new ( new CheckPoint(950, 10, 0xff72554b), new CheckPoint(10, 10, 0xff462b1d), ) ), ); Func
The main bot loop:
var startButtonPoint = new Point(950, 430); var startMissionPoint = new Point(600, 750); for (; ;) ( try ( var bmp = GetScreenImage(gameScreenRect); var screenName = check(bmp); Console.Write(screenName + new string(" ", 20) + new string("\x8", 40)) ; switch (screenName) ( case "main": mouse_event(MouseEventFlags.MOVE | MouseEventFlags.ABSOLUTE, startButtonPoint.X * 65536 / game_width, startButtonPoint.Y * 65536 / game_height); System.Threading.Thread.Sleep(400); mouse_event (MouseEventFlags.LEFTDOWN, 0, 0); System.Threading.Thread.Sleep(150); mouse_event(MouseEventFlags.LEFTUP, 0, 0); System.Threading.Thread.Sleep(50); System.Threading.Thread.Sleep (400); break; case "mission": mouse_event(MouseEventFlags.MOVE | MouseEventFlags.ABSOLUTE, startMissionPoint.X * 65536 / game_width, startMissionPoint.Y * 65536 / game_height); System.Threading.Thread.Sleep(10); mouse_event (MouseEventFlags.LEFTDOWN, 0, 0); System.Threading.Thread.Sleep(150); mouse_event(MouseEventFlags.LEFTUP, 0, 0); System.Threading.Thread.Sleep(50); break; case "action": mouse_event(MouseEventFlags.LEFTDOWN, 0, 0); System.Threading.Thread.Sleep(150); mouse_event(MouseEventFlags.LEFTUP, 0, 0); System.Threading.Thread.Sleep(50); break; case null: bmp.Save("unknown.bmp"); break; ) ) catch (Exception exc) ( Console.WriteLine(exc); ) )
During the game phase, the bot constantly clicks, releasing balls at one point. Using such simple (or rather stupid) tactics, the bot gains 1000-2000 points in the first mission, and sometimes even completely gains the Zuma bar.
Summary
The goal has been achieved: the bot framework has been written - the gameplay is looped.The following goals: connect OpenCV, recognize the position and color of the balls.
Loading...
