Using “sysbench” to test memory performance

Sysbench is a powerful testing tool for CPU / Memory / Mysql etc. Three years ago, I used to test performance of MYSQL by using it.
Yesterday, I used Sysbench to test memory bandwidth of my server.
By using command:

It reported the memory bandwidth could reach 8.4GB/s, which did make sense for me.
But after decrease the block size (Change 1M to 1K):

The memory bandwidth reported by Sysbench became only 2GB/s

This regression of memory performance really confuse me. Maybe the memory of modern machines has some kind of “Max limited frequency” so we can’t access memory with too high frequency?
After checked the code of Sysbench, I found out its logic about memory test is just like this program (I wrote it myself):

But this test program cost only 14 seconds (Sysbench cost 49 seconds). To find out the root cause, we need to use a more powerful tool — perf:

They have totally different CPU cache-misses. The root cause is because Sysbench use a complicate framework to support different test targets (Mysql/Memory …), which need to pass a structure named “request” and many other arguments in and out of execution_request() function many times in one request (accessing 1K memory, in our scenario), this overload becomes big when block size is too small.

The conclusion is: don’t use Sysbench to test memory performance by using too small block size, better bigger than 1MB.

Ref: by Coly Li ‘s teaching, memory do have “top limit access frequency” (link). Take DDR4-1866 for example: it’s data rate is 1866MT/s (MT = Mega Transfer) and every transfer takes 8 bytes, so we can access memory more than 1 billion times per second, theoretically.

Install CDH(Cloudera Distribution Hadoop) by Cloudera Manager

These days I was trying to install Cloudera-5.8.3 on my centos-7 machines, and here are some steps for operation and tips for trouble shooting:

0. If you are not in USA, the speed of network for accessing Cloudera Repository of RPMS(or Parcels) is desperately slow, thus we need to move CM (Cloudera Manager) Repo and CDH Repo to local.

Create local CM Repo

Create local CDH Repo

1. Install Cloudera Manager (steps)

2. Start Cloudera Manager

But it report:

In centos-7, the solution is:

Also need to run “sudo ./cloudera-manager-installer.bin –skip_repo_package=1” to create “db.properties”.

3. Login to the Cloudera Manager(port: 7180) and follow the steps of Wizard to create a new cluster. (Choose the local repository for installation will bring favorable fast speed 🙂

Make sure the hostname of every node is correct. And by using “Host Inspector”, we can reveal many potential problems in these machines.

After tried many times to setup cluster, I found this error in logs of some nodes:

The solution is simple:

and restart Cloudera Manager Agent on these nodes.

I also confronted a problem that installation progress has hanged on this message:

There isn’t any process of “yum” running in the node, so why it still acquire installation lock? The answer is:

4. After many fails and retry, I eventually setup the Hadoop Ecosystem of CDH:


cloudera

When upgrading or downgrading a Cloudera Cluster, your may see this problem:




The solution is (if in ‘single user mode’):

and try it again.

When staring ResourceManager, it failed and report:

The reason of this error is: there is a Non-Cloudera version of zookeeper installed on the host. Remove it and reinstall zookeeper from CDH, the yarn-resource-manager will be launched successfully.

If meet “Deploy Client Configuration failed” when create new service, just add sudo nopassword to cloudera-scm user.

Using Pig to join two tables and sort it

Having two tables: salary and employee,we can use Pig to find the most high-salary employees:

The result is: