Category: Uncategorized

Functional Tests with Jenkins, Selenium, GeckoDriver and Firefox

I’ve being going through the django testing goat book, specifically the section on continuous integration. Running you functional tests on jenkins can be quite difficult and you’ll need to tinker around and read the logs to figure out whats what. Here are a few tips.

What is the optimal versions?

Well Jenkins should be the latest because vulnerabilities are always being discovered. As for the rest there are specific compatibility issues between them but I have found the best thing to do is use this rule of thumb: Use the latest.

And I mean the absolute stable latest not just the latest for you distribution.

In my case I was on debian 8 which comes with the iceweasel firefox alternative which was at version 52.4 so I went ahead and got geckodriver and an older version of selenium for the tests. Stupid move. Mostly because you want to be up to date with real users.

Best thing to do is head over to geckodriver releases and install the latest. Get selenium 3.7 or latest and the install the latest stable firefox from firefox releases. For me that was 56.0.2.

Make sure that all the binaries are in your path and check --versions

You can install xvfb

Where Do I look when I get error?

You can check the output in the terminal when you run your test or look at geckodriver.log which I have found to be most useful.

So here are a few errors you might see:

NS_ERROR_SOCKET_ADDRESS_IN_USE

geckodriver by default runs on port 2828 and selenium runs on 4444

So they are probably already running and you need to ensure the processes are destroyed when you tearDown in your tests.

To find the process use:

lsof -l | grep 4444

and destroy it with kill -9 <process_id>

Unable to find a matching set of capabilities

selenium.common.exceptions.WebDriverException: Message: Unable to find a matching set of capabilities

This means you are setting the wrong firefox binary, make sure firefox is installed and in the path.

binary = FirefoxBinary('/usr/bin/firefox')

Message: connection refused

selenium.common.exceptions.WebDriverException: Message: connection refused

This error happens when firefox does not have a display so ensure xvfb is installed and you are starting it in jenkins

geckodriver-selenium-firefox-jenkins-errors

Conclusion

So the headless firefox, selenium, geckodriver and jenkins setup can be tedious but once it is setup you will have a much more robust CI pipeline and ensure that you end users can function onthe site.

Summary of 97% Owned Financial Documentary

97% Owned

Seigniorage – a form of fund raising for the government by selling currency to commercial banks.

It created bank notes for 4 pence, and sells the note to bank for 10 pounds, the profit goes straight to treasury and reduces tax burden.

In 10 years the Bank of England raised 18 billion pounds

In 1948 notes and coins were 17% of total money supply

Now notes and coins make up less than 3%. The rest of the money is digital and imaginary…97% owned.

Most money is now digital so it is not the central bank that creates the money it is the commercial/private banks that create the vast majority and decide how and to whom it is loaned to.

Banks create money, they don’t lend it. When you get a loan, the bank just pretends you have deposited the money. It has to invent the liability.

If everyone starts saving, the amount of money in the economy shrinks. We have a recession.

Whoever creates the electronic money gets the proceeds. It is much more profitable than creating cash in the form of notes and coins as with digital money there is zero expense.

Commercial banks have created 1.2 trillion pounds in same time it took the central bank to create 18 billion pounds with hard currency.

Banks create new money by extending credit, buying an existing asset or by making payments on their account.

When a bank buys a company’s bond it adds the bond to it’s assets and increases the company’s deposits by the corresponding amount. In other words, the bank just types in the figure it just bought the bond for on the company’s account and it has acquired the asset. So it has created new money to the value of the bond out of thin air.

People think their personal or household economy works the same way as the national economy.

That is incorrect.

The money is distributed based on the priorities of the banking sector.

If you let bankers control money supply, they will keep creating it. Why would you stop you are creating it from thin air? And it is their prerogative to acquire more loans.

Until there is so much debt that it can’t be paid back.

Money in the current system is debt. So the only way we can have money is if we have borrowed it all from the banks.

We think money is created from hard work, from working in a job. But in reality you would never have got that job without a loan / credit in the first instance.

Then people get over-indebted and cannot repay their debt.

Banks go insolvent and stop lending which causes a recession. People lose jobs and become more indebted to the banks.

If we didn’t bail out the banks it would be a total and complete killer of economic growth or the whole economy, but now there is more debt from bailout.

The only way to stop this is for banks to stop creating money. Private profit seeking banks creating 200 million pounds a year and pumping that into the economy. These private profit seeking banks are putting money into housing bubbles making houses more expensive, making their loans bigger and making more money out of thin air.

Central bank reserves is an electronic version of cash (not the imaginary numbers or bank money the general public use), it is how banks pay each other.

A High street bank will create a bond which is effectively government debt and give it to the central bank and then the central bank will type some numbers into the account for that bank at the bank of england.

So the central bank is creating these reserves out of nothing.

Before the credit crisis if a bank was short of central bank reserves, it could loan reserves from other banks with interest.

When transactions take place they use special money central bank reserves, so you buy a house from another bank they tell central bank to change values of reserves.

If they don’t have enough central bank money, then they can’t make payments and the whole system seized up. So bank must ensure this doesn’t happen.

Banks were allowed to set their own reserve targets each month.

Quantitative easing, is the process of giving commercial banks the reserve currency for free.

So central reserve money is considered real money, but fact is banks can have as much of this as they want now. It is also FIAT money, backed by nothing.

History of money

After world war 2, the UK and USA came together to manage world economies with the IMF and the world bank. At that time there was still a gold standard – Dollars was pegged to gold. And all currencies pegged to the dollar so long as americans played the roll as oversight. Preventing countries not being able to pay their bills / currency collapses. Then Americans started inflating the value of their own currency (To pay for vietnam war).

The French got worried and sent a gun boat to ask for their gold back

FIAT currency – medium of exchange where issuer does not promise to redeem in a commodity and holds its value based on confidence alone.

We believe it is worth something.

Growth and Inflation

A growing economy required growing debt.

Politicians (and many finance and economic professionals) do not realise this.

Money supply can be used to drive growth but it can also be used to inflate asset prices and for market speculation.

Inflation is the general rise in prices of goods and services. It means that each unit of currency is worth less as time passes.

When money supply grows there is more money for investments and growth but there is also more money for market speculation and buying of goods.

Inflation is caused by too much money chasing too few goods and services. When money supply is growing at a faster rate than goods and services.

Recorded / Measured is flawed

CPI is a measure of the increase in price of  basket of goods and services over time. It is deemed to provide a consistently lower figure for inflation. This is because house prices, mortgage repayments and council tax are excluded from the calculation.

RPI retail price index is deemed a better representation of inflation.

The biggest expenditures one makes should be taken into account, like house or car / school fees.

The increase in mortgage/loan on a house does not increase the economic output of a naction. It just increases money supply and hence does not enhance GDP, causing inflation. Banks creating money leads to more speculative credit and higher valuations on safe assets.

You can give a loan to a small business, is more risky as there is less collateral. Giving a Loan to a house, on the other hand, there is collateral. Not productive investments.

Bubbles

High inflation on a specific good or service.

The Tulip mania. The money system is not abstract it has alot to do with nations, power, trade and how they interact.

The ideal attributes for bubble creation: Luxury and Necessity.

Inflation can be avoided if money supply or creation does not exceed the economic output

Argument for government to guide where money should be invested (war economy)

People are getting poorer all the time, money is distributed from the poor to rich.

Every pound of money, has a pound of debt.

Debts from the poor to the rich are set in stone and are now sacred.

The reason the poor are in debt is because the prices have gone out of control and when the system breaks the poors are the ones owing.

Bank Run

You can withdraw all your cash from bank, but this does not reduce digital money supply.

You can stop the monopoly by moving your money into local community banks, not these massive private banks.

International bank run – withdraw from one currency to another, reserve currency shifts from reserve currency to international bank. But not part of their local central bank.

Currency Wars

It will get a trade imbalance.

Spending more than they are earning = trade deficit. Ability to repay debts is questioned. You can devalue your currency so exports increase. Domestic industry demand has grown.

Central bank can sell reserve currency in the market to devalue currency (this reserve is created from nothing, typed into a computer).

Belief is the thing holding up a currency.

Third world debt is used as a form of colonialism, having power of the economy controlling what they do.

IMF tells 3rd world countries that they can pay back their debt by increasing exports so they are earning more dollars so you can pay off your debt. Which is all a lie.

In reality countries cut their government spending and hence they stop growing. So they paid their debts but their own economy was not being developed. So the country becomes poorer and then big corporations come to exploit its natural resources.

These rules imposed by IMF actually destroys local industry and makes more dependent on foreign loans.

Also tell countries to lower tax in multinational corporations.

Also means profits made in country go out and do not help locals.

To manage risk on this unbacked currency you needed derivatives, futures and new markets. Hedging = insuring against your risk.

Derivatives based not on real products were essentially gambling, which changed in the 1960.

Efficient market hypothesis, The theory is that a market regulates itself better than if a government interferes.

The 2008 credit crisis caused that belief to end. Anyone who still believes the market is self regulating is a pencil neck.

Credit default swaps – insurance against companies from going bust – inflated from 1 trillion to 60 trillion in 5 years. But it turns out they don’t provide stability and the maths inside them is completely borked.

Cash is backed up by government debt and government debt is backed up by the ability of government to get money from the public through tax.

System is designed to make a few people very rich at the expense of taxpayers and citizens.

It Lowers standard of living of majority.

Currency Reform

So what can we do…

One hypothesis it to back a currency by renewable energy, which will increase investment in that space.

Banks should have to ask you what they do with your money. They shouldn’t be able to gamble with it.

A safe account and an investment account so banks don’t need to be bailed out by government.

Person to person banking.

We should not ask the banks for advice on how to improve the money system, they are the last people to ask.

You wouldn’t ask a bad house builder advice on how to build a house.

CPU usage high…is there a bot flooding you with requests?

CPU Usage High Is there a Bot at work?

Alright something is going on, we are getting new relic alerts.

cpu-usage-high-new-relic

As you can see something strange started on the 5th of October. Check Under server -> Apps on New relic and see what is the application that is causing the High CPU Usage. In this case it was apache at 80.4% CPU usage.

The first port of call would be the /var/log/apache/access.log and if you tail -f the log you will see the frequency of requests. An example is shown below.


66.249.64.135 - - [14/Oct/2016:12:10:45 +0000] "GET /products/new-products-category.html?color=white,ivory,brown,beige,natural&dir=desc&order=position HTTP/1.1" 403 587 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.64.135 - - [14/Oct/2016:12:10:47 +0000] "GET /products/grass-products-category.html?color=pink,blue,natural&dir=desc&order=position&p=2 HTTP/1.1" 403 585 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
41.72.195.90 - - [14/Oct/2016:12:10:46 +0000] "POST /index.php/autoassign/adminhtml_api/index/key/1be7da78e61ac5c2f9f7ed4e16084a22/?isAjax=true HTTP/1.1" 200 744 "https://www.example.com/index.php/admin/catalog_product/index/store/0/key/7d2daff670207536c026ba902cd4dce6/" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.143 Safari/537.36"
66.249.64.135 - - [14/Oct/2016:12:10:48 +0000] "GET /product-decor/mosaic-listellos-category.html?color=gold,red,black HTTP/1.1" 403 595 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.64.135 - - [14/Oct/2016:12:10:50 +0000] "GET /products/stonewall-dabbing-category.html?color=black,blue,light-brown&dir=desc&order=position HTTP/1.1" 403 592 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.64.135 - - [14/Oct/2016:12:10:51 +0000] "GET /products/new-products-category.html?color=natural,grey,beige,white,brown&dir=desc&order=name HTTP/1.1" 403 587 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.64.135 - - [14/Oct/2016:12:10:53 +0000] "GET /product-decor-category.html?color=gold,white,ivory,pink,grey HTTP/1.1" 403 578 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.64.135 - - [14/Oct/2016:12:10:54 +0000] "GET /products/new-products-category.html?color=grey,ivory,natural,brown,beige&dir=desc&order=name HTTP/1.1" 403 587 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.64.135 - - [14/Oct/2016:12:10:56 +0000] "GET /products/stonewall-dabbing-category.html?color=white,bronze,terracotta&dir=desc&order=position HTTP/1.1" 403 592 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.64.135 - - [14/Oct/2016:12:10:57 +0000] "GET /product-decor-category.html?color=gold,red,white,black,pink,grey&dir=asc&order=name HTTP/1.1" 403 578 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

What else can we do to check bots

You can also use:

To check the number of requests in that many seconds.


tail -n 500 /var/log/apache2/access.log | cut -d' ' -f1 | sort | uniq -c | sort -gr

First Steps

The first thing to do is install fail2ban on the server and configure it for apache.

We have found the Issue

There is an ip: 66.249.64.135

Is making lots of requests to pages that request a lot of processing. What is more is it says it is Googlebot/2.1; +http://www.google.com/bot.html…Yeah right.

Solving it

First Port of Call

Block the ip with .htaccess:


Order Deny,Allow
Deny from 66.249.64.135

Next Steps the automatic solution

So for the complete solution we need to block IP’s that are making more than 300 GET requests in 300 seconds. Note you should change this based on your criteria.

Add this to jail.local:


[http-get-dos]
enabled = true
port = http,https
filter = http-get-dos
logpath = /var/log/apache2/access.log
maxretry = 300
findtime = 300
#ban for 5 minutes
bantime = 600

This will check your apache access log and apply the http-get-dos filter to it.

In the filter.d directory do the following:

Do vim http-get-dos.conf:

then add the following in there:


# Fail2Ban configuration file
#
# Author: http://www.go2linux.org
#
[Definition]

# Option: failregex
# Note: This regex will match any GET entry in your logs, so basically all valid and not valid entries are a match.
# You should set up in the jail.conf file, the maxretry and findtime carefully in order to avoid false positives.

#failregex = ^ -.*GET.*/ip\.cgi
failregex = ^ -.*"(GET|POST).*

# Option: ignoreregex
# Notes.: regex to ignore. If this regex matches, the line is ignored.
# Values: TEXT
#
ignoreregex =

Yeah so this should do the trick. I have found that if you specify an action, it won’t actally block that ip.

I will update with results.

The .htaccess change seems to have done the trick:

blocking-an-ip

When CPU usage was low, that was when the .htaccess was edited.

Turns out it is a REAL google bot

To check if the bot is a real google bot check this link. Strange that it is spamming us silly.