Incapsula Blog

Here's a thought experiment. This one requires some long-term thinking, outside the box and well past recent history and the status quo.

What if the majority internet usage is non-interactive, from so-called "bots", what we may refer to as "automated use". Google and Facebook, among others, rely on the use of automation and "bots". The non-interactive clients ("bots") being used by these companies are not asked to solve captchas. (In turn, after collecting data from public sources, these websites attempt to prohibit the use of automation by their users wishing to access it. What is interesting is that neither company provides any definition of "automated" nor any clearly stated limits on the speed at which a user may access resources or the quantity of resources they may access in a stated time period. One might be apt to find such limits associated with an "API".)

In 2013 an Incapsula report suggested that the majority of internet usage is in fact automated and not "malicious"^1 -- what if public information sources on the internet catered to the use of automation rather than trying to limit such use, e.g., with speed bumps^2 like "captchas". What if servers treated all clients equally, instead of having data forcibly collected by a few large clients that receive preferential treatment, then siloed and protected from "automation". What effects would this have on "centralisation" and levelling the playing field.

"Do not ask for permission, ask for forgiveness." …

I’ve been working on a bot monitoring platform - hoping to launch soon.

Problem:
Up to 61.5 percent[1] of website visitors are bots, of which 30.5% are malicious. The presence of bots lead to content theft, malware injection, server hijacking, spam links and DDoS attacks. Website owners are often unaware of their existence and the extent of their effects.

Solution:
A platform to monitor and analyse bot traffic. We alert you to attacks, pinpoint affected pages and identify and analyse all bots. To retrieve traffic data we use pixel tracking & JavaScript tracking code. To detect bots we look for abnormalities in a series of traffic data such as HTTP header fingerprinting, JavaScript footprint, timestamps, IP reputation etc.

With the results from our platform you can better implement methods of prevention such as blocking IP addresses/user agents or go about including CAPTCHAS or similar where necessary.

If this is something you would be interested in I’d love to hear what you think :).

[1] http://www.incapsula.com/blog/bot-traffic-report-2013.html

Hi,
I work for Incapsula. Yesterday we've addressed this study in our blog.
http://www.incapsula.com/the-incapsula-blog/item/699-incapsula-pentested-review

To make a long story short, this detailed report helped us introduce several patches to our WAF - but we feel that some of the issues were mostly theoretical and patching them can lead to security eroding false positives.

ModSecurity is a great security solutionand we are happy to be compared to it. We feel that we perhaps offer a more complete package (bot filtering, acceleration, user friendly GUI and easy setup, support, etc).

Still, you can't beat Free and if you are looking for OS security solution, ModSecurity is defiantly a name you should consider.

On a personal note, I wish that more vendors would take security a bit more seriously - especially if they claim to provide it to thousands of clients.

Web evolution is motivated by consumer trust and if we loose this trust we will also loose the cash flow that keeps Internet evolving.

Providing faulty security will do just that.

I wondered how they distinguished bots. The only info I found on the study [1], on a quick search, says:

"For the purpose of this report we observed 1.45 Billion bot visits, which occurred over a 90 day period. The data was collected from a group of 20,000 sites on Incapsula’s network"

So, in the first place, not necessarily representative of the web generally. And it does not say how they detect bots. Of course some declare themselves [2]; otherwise I guess one would have to rely on captchas [3], which are known to be of limited reliability and subject to an arms race.

1. http://www.incapsula.com/the-incapsula-blog/item/820-bot-traffic-report-2013

2. kristopolous on this page, '(bot|spider|crawl|aggregator)'

3. which Incapsula does, as a condition of access to its summary of its previous similar research.

> Per the article

Not quite, it says "If the attacker has a bigger fire hose of data than the defender has, the attacker wins" and "the size and scale of these probes—and especially their persistence—points to state actors" which is not quite the same as saying you need to own the bandwidth. For example, DNS amplification can be used "to turn initially small queries into much larger payloads, which are used to bring down the victim’s servers".

https://www.incapsula.com/ddos/attack-glossary/dns-amplification.html

So maybe there are other techniques which might allow for similar leverage. Neither is the article conclusive about "state actors", they are merely "pointed to". As for expertise ... I don't doubt there are people out there who have it or might acquire it. So it's still an interesting question imo.

So many devices are now connected to the Internet and potentially vulnerable. The basics definitely matter – changing default passwords, ensuring our internet is hosted on DDoS protection servers, etc. But who was going to warn the traffic engineer that their security camera is vulnerable or the new parents whose IP-connected baby monitor gets scanned by foreign hackers. We just want things to work and don’t realize that we’re at risk – even if our device is the target and not ourselves.

Have you checked out this Mirai vulnerability scanner? Something everyone should do – whether a random home user or a large enterprise (and how many have CISOs?). It scans your IP and can pinpoint vulnerable devices: https://www.incapsula.com/mirai-scanner.html