Category: Database

Going Live with Django, MySQL on Ubuntu and server going down

Ever got to deploy day only to have errors streaming in about the site not being able to be accesses. Most likely in your logs or error emails / alerts you get something like this:

(2003, "Can't connect to MySQL server on '127.0.0.1' (111)")

What the hell does this mean, it means MySQL is down and if it was running it means it has crashed. Whoah but if you check the site now it is working…that means nothing the website is unstable as certain actions make it go down. We need to achieve system stability.

The most common reason for MySQL going down is that it is using more memory than physical memory can allocate it then uses virtual memory and crashes soon after that.

What can I do?

First double check the error logs and see if there is a configuration error causing this issue. If there is no clear warning you want to check memory usage.

Check if you are using virtual memory with vmstat


procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 1  0      0 499228  72040 358540    0    0    16    27  230  280  2  1 97  0  0

Check the amount of memory being used:


ps aux | grep mysqld

A few good commands to use during the process:

In mySQL:

show global status;

Source:

Best Practices for Configuring Optimal MySQL Memory Usage

Comparing Providers for the JSE stocks, commodities, futures and currencies

I have needed End of day and sometimes live data for my trading activities and other similar endeavours and it is hard to find a data provider. Particularly if you want stock data on the JSE as this data has become a commodity for them and they want exorbitant fees for the service.

So I have checked out what is available and I’ve found the following options:

  • quandl – A aggregator of stock data providers. It seems like they provide the platform and API and resell data from other data providers.
  • tiingo – Free end of day data.
  • eoddata – A pay-for service providing downloadable stock, commodity and other such financial data. An old looking site and no API (oh no…)
  • eodhistoricaldata – A pay for service that provides pretty much everything you need

EOD Data

Check the symbols they provide here