Solving Intermittent WordPress Database Connection Errors

A week ago, my website suddenly started becoming non responsive. Pages would take forever to load and going to the dashboard to post was a nightmare, and worst of all, I was starting to get inundated with “500 coded” database connection errors. I place a lot of emphasis on keeping my site running fast and smoothly and delays of this sort were disastrous for me. I was at my wits end.

I have a shared hosting plan with Bluehost and also use the free Cloudflare service. So not a lot of resources at my disposal either.

Errors of the sort shown in the screenshot below would often disappear if I refreshed the page. I tried just “waiting it out” figuring that Bluehost was having issues at their end, but it continued on for several days. At times my blog would become completely unresponsive and just sit there trying to connect and would then show an error page.

Later, my server logs revealed that I was getting around 1500 error results on my site every day! That meant legitimate visitors were being forced to wait for long periods of time and ultimately not getting the content they came for.

Sudden WordPress Database Error
Sudden WordPress Database Error

 

The WordPress forums didn’t help. They all dealt with issues like the password combo being wrong, or replacing “localhost” with the name of the server in wp-config.php. But none of these account for the fact that the errors were random. If there was really a configuration problem, they wouldn’t vanish on a page refresh. Besides, I’ve been maintaining this blog for several years now and there’s never been anything of this sort before. I hadn’t changed anything.

So I called Bluehost to complain. After the woman on the other end spent time consulting with the tech person, she told me that my site was getting overwhelmed by visitors. The shared plan I was on allowed only 15 database connections per second. She told me to be happy that I had outgrown the plan I was on! I told her this wasn’t possible because my Google analytics wasn’t showing any such dramatic increase in visitors. She then visited the stats page and there we found the problem.

My site was swamped with spam and bots. By half the month, one single unknown bot had eaten up 1.5 GB of bandwidth and had hit my site 29,000 times! In addition, Bluehost has a CPU throttling facility where if a site uses up too much CPU power to the detriment of the others, it’s cut down. That screen was showing me that my CPU was being throttled for around 60,000 seconds a day – meaning almost all the time! Well, at least now that I knew what was wrong I could try and do something about it.

I had become complacent about traffic and bot management on WordPress because I use Cloudflare and they’re supposed to block out most undesirables. Unfortunately, I had found that even the “medium” security setting was blocking many legitimate users of my site who had complained, so my filter was on “low”. Clearly this wasn’t doing a good enough job. So I temporarily put it on high while I sorted out the issue at my end.

To no avail. The bots kept coming, and now that I knew what to look for I was shocked at how aggressive they were. My site was completely inaccessible and I had to switch on the “I”m under attack!” mode in the Cloudflare control panel where all visitors got a 5 second delay and turned aside if they were bots. I had to do this to access my site while I tried to fix things.

“Bad Behavior”, “Better WP Security” and “ecSTATic”

I noticed that my “wp-login” page was being hit the most. I had to beef up my security. So two plugins came to the rescue. “Bad Behavior” is one that modifies your .htaccess file to block all kind of baddies at your doorstep. It also allows you enter your Project Honeypot key so that all known visitors with a threat level of 25 and above are stopped. I had to sign up with Projecty Honeybot to get a key and also enabled “Strict checking” in the plugin. I’m not sure what that option does, but it blocks more malicious visitors.

More important than Bad Behavior though was a plugin called “Better WP Security” that really hardens your site. Among other things it changes the admin and login page locations to something else, thus denying bots convenient attack locations returning a 403 instead of wasting bandwidth on your site even if they don’t succeed. This alone saved my site from thousands of hits. But the plugin also does a LOT more from removing the default “admin” user, to allowing you to ban bots and users from your site and also scans it for security vulnerabilities.

The final plugin is called “ecSTATic”. This is a very powerful plugin for banning bots from your site. There is an option called “WTF” or “Way too fast” which you configure to ban any bot that makes too many requests within a certain period of time. Plus you can also block all unknown bots which is fairly safe I guess since the “known” list is pretty extensive. And you can configure detailed rules for denying access based on user agent or IP addresses.

Banning bots like this is necessary if they don’t follow robots.txt as described in the next section.

Robots.txt

I had ignored my robots.txt for quite a while since I believed that cloudflare would filter out misbehaving bots. But it didn’t. Even at the “high” security levels, cloudflare doesn’t do a good job of protecting your site from bad behaving bots that eat up your bandwidth and hit your database. It’s up to you alone.

So the first thing you do is to check your server logs. Every site has a few irritating spiders that hit it too often. My bane during this difficult time was “80 Legs” and “008 AhrefsBot”. So I disallowed them in my robots.txt file like so:

User-agent: 008
Disallow: /

user-agent: AhrefsBot
Disallow: /

Unfortunately, only 80 Legs followed robots.txt. AhrefsBot simply ignored it. That’s what bad bots do and they deserve to get their ass banned either using the plugins I showed earlier, or using .htaccess. This practice alone has relieved my site of a ton of traffic.

Blocking URL Parameters and Archive Pages

Going over the list of pages crawled by Google and others, I found that a whole ton of useless URLs were being crawled. As a WordPress blog, all my posts have multiple “replytocom” URL parameters. One for each comment I think. And these were being crawled by Google – around 7000 pages worth! And I have just  464 unique posts. The solution is to block these URL parameters from being crawled using robots.txt. So for me, the relevant entry was:

Disallow: *?replytocom

This puts a stop to a lot of crawls that were eating up my bandwidth and unnecessarily hitting up my database. There’s also an option in Google’s Webmaster tools that can let you specify what URL parameters you don’t want them to crawl. But the robots.txt entry is more elegant if you have control over it. Otherwise, the webmaster tools section will do just fine – but only for controlling Google crawlers naturally.

End Result of the above steps

My efforts paid off. In the next few days, my “500 code” database errors disappeared to almost nothing and my website loading speeds went back to normal levels. I kept a close watch on my stats and visitors as well as the Bluehost throttling panel for the next week to catch any relapses and ban any more misbehaving bots. But so far, it’s been pretty smooth sailing.

I saved my WordPress blog of six years for free and improved my security to boot!

What do you think of this post?
  • Agree (3)
  • You're an asshole (1)
  • Don't Agree but Interesting (0)

14 thoughts on “Solving Intermittent WordPress Database Connection Errors”

  1. Hi Bhagwad,

    Thanks so much for your post. I’m having the same problem, nice to hear someone say something besides “check your wp-config file.” I enabled the logs with network solutions (my hosting provider). Here is the log so far, from the last 10 minutes or so: http://santamonicapier.org/files/newsletter/access_log.2012-10-03

    Would you mind having a look and letting me know if you see any potential concerns for spam or bots? I’m a front-end developer and don’t know that much about these things.

    Either way, I will employ the suggested plugins.

    Thanks so much!

    Ivan

    Reply

  2. Hey I face intermittent database error as well and you have written a great blog! I am still not quite sure if my site is under attack by bots. I would like to check will this block google spiders from indexing my pages as well? Cheers!

    Reply

  3. Thanks so much. I had the same exact problem with my bluehost shared hosting, but unfortunately it took 4 long phone calls over four days and they still couldn’t easily diagnose it as was your experience with them. I used the plugins you recommended and the site came back up correctly and consistently right away.

    THANK YOU!

    Reply

  4. Hi, your recommendations helped us a tremendous amount! Thanks, we installed these plugins and we are back in business. I am looking fr advice on how to configure ecSTATic . I suspect it is now blocking search engine bots as well. BingBot seems to be an extremely frequent bot that hits our server.

    Reply

  5. I’ve installed these plugins; already had installed “Better WP Security”. Looking forward to seeing if my site survives the “error connecting to database”. Checked wpconfig, htaccess and all the details inside on other sites to no avail.

    This is the first site that gives another option. Thanks. I’ll keep you posted next week, as I was receiving this message every day, if I don’t anymore then this solution has done the job.

    Reply

    • In reply to David K

      Okay so, just after posting the previous post, my site came to a crawl, then I got the dreaded “error connecting to database”. So it’s something else. Thanks anyway. I’ll have to investigate alternatives

      Reply

  6. Thanks Bhagwad,

    This post saved my day as my site went down in the morning and looking at the db connection I had the error of too many connections. So I closed my site to allow the database to cool down and implemented all the advice in your article and now have a fully restored site.

    I am keeping my eye on the logs again and this will teach me not to be too complacent about security and those plague of the internet ‘bad bots’.

    Reply

  7. I am the web manager for this website and I inherited a wordpress installation that was full of junk, garbage, nastiness, and ZERO SECURITY and I tired so many things to fix this very issue and this article and those plugins helped everything. You are a godsend, and I wish I could embed a six-pack of your favorite brew in this comment box because I am so thankful to have this behind me. Thank you so much!

    Reply

Leave a Comment