I'm curious whether we'll hit a point where some sites will want to be as friendly to bots as possible (hopefully they'll just shift to having sane, rate-limited public APIs so everyone can save time and bandwidth), while the rest want the reverse.
If you're providing a service, and AI agents truly take off to where people say "Claude, go buy me a new shower hose about 5' long, brushed nickel. Use my usual evaluation criteria and don't spend over $20." then a lot of ecommerce stores are very very incentivized to be bot friendly.
And of course every "content" site is the reverse, wanting to not waste their bandwidth and hoping to score exclusive paid content deals with any entity that might want to scrape them.
I was experimenting with Google Ads traffic to one of the websites I cared about, and even just looking at the existence of mouse move and scroll events, let alone the patterns, made me understand how many bots, or users who just (mis)clicked the ad and never did any action on my website, were counted as paid clicks in Google Ads. Analyzing for patterns like in this article would've made even more sense.
I alway wondered why so many people (not saying you're one of them) get mad at being charged for misclicks and bots.
It all averages out.
You bid $3 a click because your conversion ratio is x, and paying $3 is worth it. Even if 90% of the traffic is bots, it works out.
If Google filtered out all the bots, the conversion ratio for you and your competitors would go up, and your competitors would be willing to pay $27 per click and you'd outbid at $30, being at the exact same result at the end of the month spent for the exact same results.
well it's funny but when I was writing my bot targeting a particular platform I did a lot of this stuff, because I was paranoid and didn't want to get blocked, so things I did:
detect all element in currently loaded page I want to do things with. element I want to click. scroll to element I want to click. random viewport scrolling action, go slightly over viewport, scroll back up, determine mouth position where I will click - x, y center of clickable item, determined x,y of bounding box of clickable item, choose some point within bounding box where I can click with some randomness for getting to center because obviously humans don't get the exact center every time do they. set random mouse move speed. mouse choppiness function (determine a number of x.y coordinates on the way towards the x.y you will actually be clicking move towards those each in turn with randomness of mouse speed, so that nobody can say huh, this mouse is moving towards the place it will click with an unnaturally direct and unnaturally smooth movement.) Get to where I will click. wait random time to click. click.
If I do not continually get new elements I want to click based on my criteria just by this method now I need to scroll to see if I can detect new.
random number of times I will scroll. random scroll speed. random scroll distance.
start scroll. hit scroll distance stop. random up down scroll.
My automation goal was not to make my bot able to do the work of hundreds of people. It was to do the work of a few people.
However I eventually gave it up since the target platform was continually deciding I, as a human, was somehow behaving as a bot, it followed that it was somewhat irrational to try to make a bot that behaved as close to a human as possible.
(note there was all sorts of other erratic behavior boiled into my bot, for example how it wrote text, since it had to do keypresses, and randomized typing speeds, and making errors, and back spacing to fix errors)
on edit: before you start telling me about how it's not randomized enough or whatever, I am of course giving a curtailed high level description of the kinds of noise I added in to my bot behavior, because I just really wanted to spin up a bot that could spend a few hours doing stupid work while I did other things, and then write up a report and then later I could come in and do some more clever things.
it should be noted that later I was told by someone who should know that I had over-thought and over-architected my bot and nobody checked for even 10% of the things I was doing to make my bot behave the way a human did.
This is another one of those horse-blinders style arguments that considers only for-profit/institutional/etc contexts and so doesn't realize their very method of measuring something through javascript program execution is biased or that it's creepy.
I'm a human, I don't execute javascript programs and by default my browser does not either. Anyone using this will label me a bot.
I'm curious whether we'll hit a point where some sites will want to be as friendly to bots as possible (hopefully they'll just shift to having sane, rate-limited public APIs so everyone can save time and bandwidth), while the rest want the reverse.
If you're providing a service, and AI agents truly take off to where people say "Claude, go buy me a new shower hose about 5' long, brushed nickel. Use my usual evaluation criteria and don't spend over $20." then a lot of ecommerce stores are very very incentivized to be bot friendly.
And of course every "content" site is the reverse, wanting to not waste their bandwidth and hoping to score exclusive paid content deals with any entity that might want to scrape them.
That's great!
I was experimenting with Google Ads traffic to one of the websites I cared about, and even just looking at the existence of mouse move and scroll events, let alone the patterns, made me understand how many bots, or users who just (mis)clicked the ad and never did any action on my website, were counted as paid clicks in Google Ads. Analyzing for patterns like in this article would've made even more sense.
I alway wondered why so many people (not saying you're one of them) get mad at being charged for misclicks and bots.
It all averages out.
You bid $3 a click because your conversion ratio is x, and paying $3 is worth it. Even if 90% of the traffic is bots, it works out. If Google filtered out all the bots, the conversion ratio for you and your competitors would go up, and your competitors would be willing to pay $27 per click and you'd outbid at $30, being at the exact same result at the end of the month spent for the exact same results.
'Clicks' like that have bolstered stock prices for decades now.
well it's funny but when I was writing my bot targeting a particular platform I did a lot of this stuff, because I was paranoid and didn't want to get blocked, so things I did:
detect all element in currently loaded page I want to do things with. element I want to click. scroll to element I want to click. random viewport scrolling action, go slightly over viewport, scroll back up, determine mouth position where I will click - x, y center of clickable item, determined x,y of bounding box of clickable item, choose some point within bounding box where I can click with some randomness for getting to center because obviously humans don't get the exact center every time do they. set random mouse move speed. mouse choppiness function (determine a number of x.y coordinates on the way towards the x.y you will actually be clicking move towards those each in turn with randomness of mouse speed, so that nobody can say huh, this mouse is moving towards the place it will click with an unnaturally direct and unnaturally smooth movement.) Get to where I will click. wait random time to click. click.
If I do not continually get new elements I want to click based on my criteria just by this method now I need to scroll to see if I can detect new.
random number of times I will scroll. random scroll speed. random scroll distance.
start scroll. hit scroll distance stop. random up down scroll.
My automation goal was not to make my bot able to do the work of hundreds of people. It was to do the work of a few people.
However I eventually gave it up since the target platform was continually deciding I, as a human, was somehow behaving as a bot, it followed that it was somewhat irrational to try to make a bot that behaved as close to a human as possible.
(note there was all sorts of other erratic behavior boiled into my bot, for example how it wrote text, since it had to do keypresses, and randomized typing speeds, and making errors, and back spacing to fix errors)
on edit: before you start telling me about how it's not randomized enough or whatever, I am of course giving a curtailed high level description of the kinds of noise I added in to my bot behavior, because I just really wanted to spin up a bot that could spend a few hours doing stupid work while I did other things, and then write up a report and then later I could come in and do some more clever things.
it should be noted that later I was told by someone who should know that I had over-thought and over-architected my bot and nobody checked for even 10% of the things I was doing to make my bot behave the way a human did.
This is another one of those horse-blinders style arguments that considers only for-profit/institutional/etc contexts and so doesn't realize their very method of measuring something through javascript program execution is biased or that it's creepy.
I'm a human, I don't execute javascript programs and by default my browser does not either. Anyone using this will label me a bot.