news_header_sep2019

Hi [subscriber:firstname | default:]

Earlier this week a number of our hosted clients reported difficulties accessing their kiwitrees site. Most were seeing something like this instead:

508
The phrase "Resource Limit Is Reached can mean various things. The full definition is "The error message "508 Resource Limit Is Reached" appears when your account is constantly exceeding the resources assigned to it - these can include CPU usage, RAM usage and/or the number of concurrent processes running". In this case, and also a similar occurrence in January is related to the last part: "the number of concurrent processes".
That means, in simple terms, too many attempts to access a site at the same exact time. Looking at the server logs we could see, on one of our largest client sites, literally dozens of visits to the site, to the same page, from a multitude of different IP addresses, at the exact same time. Further investigation showed that the site had received 757,000 access requests (visits) in the last 10hrs. This is over 21 requests per second on average. Our servers have generous limits for all resources, but not to this level!
This caused that site, and then any others on the same server, to lock up for the duration of each attack, as you experienced.
This "attack" was mainly from IP addresses emanating out of China (although plenty of other countries can and do cause similar issues), sending out web crawlers, or "robots" to gather every possible link from as many random web sites as they can. This is termed "SEO (search engine optimisation) spam". It is designed with purely commercial interests in mind - gathering millions of links, then selling them to paying advertisers.

Family History websites, by their very nature, contain a lot of links. Look at any individuals' page on your site to see how many there on just one page.

Kiwitrees already has in place systems to block robots from any page that has large numbers of links that provide a search engine with no value above that already available on basic individual pages. This system blocks pages like the calendar, lists, charts, reports, etc.. But it can only block robots that follow the rules. Robots from the likes of Google, Bing, Microsoft, Facebook, etc. know the rules and follow them. These newer spam robots do not!

To combat these new robots we are upgrading the levels of security, and improving our ability to block them, even to the extent of blocking entire countries if necessary. This has been applied to the client site most involved in this latest attack. The result was an almost instant end to the problem. Further 'tweaks' are ongoing, and getting close to complete eradication - although 100% is never likely to be achieved.
Once we are satisfied we have the correct measures in place, similar changes will be applied to all kiwitrees clients.
This may involve a brief period (maximum a few hours) when your site will be unavailable, as we redirect it to the new environment. You will be notified when that is likely to happen, sometime over the next week to ten days. We apologise for any inconvenience.

Finally - the good news.....
  1. This, and similar attacks are annoying and inconvenient, but they do NOT risk your data in any way whatsoever. At no time do this sort of attack gain access to your files or database.
  2. The new environment, using "Cloudfare.com" servers may result in slightly faster page loading. This is due to it's automated resource caching on a globally distributed network, meaning fixed information such as javascript files, style sheets etc will be stored and sent to your browser from servers closer to you than New Zealand.
nigel_sig
twitter