Dealing with Connection Timeout to Redis when the "Reclaimed Items" Spike

Recently I encountered a Redis problem, where new Redis connections failed to be established from the server, and the client massively timed out. The AWS managed Redis cluster has a few metrics coincide with the surge of connection issue: "New Connections" and "Reclaimed Items" both spiked during the time of difficulty.

Commonly Redis would respond slow, causing connection issue on the client when "Eviction" events occurs. This usually happens where the max capacity of the Redis node has been reached, and entries are forced out of Redis as a result. However, in my case the "Eviction" metrics showed 0, and "Bytes Used for Cache" suggested that Redis has enough RAM. Clearly, the surge of "New Connections" is a symptom rather than a direct cause. When the system was halting, normally connections would be expected to build up waiting on a certain resource.

The AWS documentation shows that the "Reclaimed Item" is the
"The total number of key expiration event". https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/CacheMetrics.Redis.html

Redis documentation explains that "basically [key] expired events are generated when the Redis server deletes the key" after TTL on a key goes to zero. https://redis.io/topics/notifications Since the background Redis process may keep deleting items when the collected sample has >25% of expired item, Redis may take a few seconds to delete when large amount of items are expiring.  https://redis.io/commands/expire This would make established connections to be unable to proceed their operations, causing connection issue for the client.

Note that this problem is related to both write/second and the TTL length. You have an option of either reducing write/second, or make TTL shorter so that the number of items don't build up at a potential of being deleted all at once by the background process.

When will Hystrix Throw ThreadPoolRejection Error when Synchronous Queue is used?

Hystrix uses SynchronousQueue when MaxQueueSize has been set to -1, and uses LinkedBlockingQueue when MaxQueueSize is set to >=0.

When LinkedBlockingQueue is used and the number of concurrent requests is beyond (threadPool.CoreSize + queueSizeRejectionThreshold), ThreadPoolRejection Exception will be thrown.

When SynchronousQueue is used and the number of concurrent requests is beyond (threadPool.CoreSize), ThreadPoolRejection Exception will be thrown.

Here's a snippet providing it when SynchronousQueue is used:

submitting:0
submitting:1
submitting:2
submitting:3
submitting:4
submitting:5
running: 1
running: 2
running: 0
running: 4
running: 3
submitting:6
get Object of 6
Exception in thread "main" com.netflix.hystrix.exception.HystrixRuntimeException: TestHystrix$1 could not be queued for execution and no fallback available.
 at com.netflix.hystrix.AbstractCommand$21.call(AbstractCommand.java:783)
 at com.netflix.hystrix.AbstractCommand$21.call(AbstractCommand.java:768)
 at rx.internal.operators.OperatorOnErrorResumeNextViaFunction$1.onError(OperatorOnErrorResumeNextViaFunction.java:99)
 at rx.internal.operators.OperatorDoOnEach$1.onError(OperatorDoOnEach.java:71)
 at rx.internal.operators.OperatorDoOnEach$1.onError(OperatorDoOnEach.java:71)
...

Note that ThreadPoolRejection starts to happen when all 5 threads in the threadPool have been taken.

Vim Tip - Delete All lines above / below

vim command to delete all lines above - dgg
vim command to delete all lines below - dG

UTF8 Encoding Problem in Laravel Blade Template

I have a Chinese site written in three years ago. It is still on Laravel 3, but I'd like to migrate it into a new server. After the migration, it appears that all Chinese characters from the application are rendered twisted. And it was not because of the <meta> tag  that can fix it, as I am seeing the wrong character in the source.

My first step was try to render some Chinese characters from plain .php file. It worked. That ruled out the Nginx/HHVM configurations. After that, I started to place "print_r()" with some UTF-8 characters. Tracing to /laravel/core.php, it appears that mb_output_handler method is using a wrong default encoding. I made the following changes:

mb_http_output("UTF-8");    // to make blade template output in UTF-8
ob_start('mb_output_handler');

The UTF-8 characters appeared normal again after the change. Great! :)

List of Python Unicode / UTF8 String Encoding Snippet

\u to \x conversion

>>> print(u'\u0420\u0443\u0441\u0441\u043a\u0438\u0439')
Русский
>>> a = u'\u0420\u0443\u0441\u0441\u043a\u0438\u0439'.encode('utf8')
>>> a
'\xd0\xa0\xd1\x83\xd1\x81\xd1\x81\xd0\xba\xd0\xb8\xd0\xb9'
>>> print(a)
Русский


\x to \u conversion

Firstly, convert unicode to string (iso-8859-1 encoding)
Then you need to know the encoding. In this example, the encoding of the string is gb2312:

>>> x
u'\xcc\xd8\xbe\xaf\xc1\xa6\xc1\xbf'
>>> x.encode('iso-8859-1')
'\xcc\xd8\xbe\xaf\xc1\xa6\xc1\xbf'
>>> '\xcc\xd8\xbe\xaf\xc1\xa6\xc1\xbf'.decode('gb2312')
u'\u7279\u8b66\u529b\u91cf'
>>> print(u'\u7279\u8b66\u529b\u91cf')
特警力量

How to Make Slow Mongo Query on Compound Index with $gte and $lte Fast

I have a mongo collection of IpRanges, with start_address and end_address. I'd like to query given an IP. with compound index on {start_address:1, end_address:1}, this however is still very slow:

db.ip_blocks.find({start_address: {$lte: 1665637698}, end_address: {$gte: 1665637698}})


with .explain(), it appeared that this query causes a lot of scanning and slow, as only the $lte query worked hard, and the second query is scanning what's remaining from the first query.

Here is a trick to speed this up if you are only trying to get the first match:

db.ip_blocks.find({start_address: {$lte: 1665637698}, end_address: {$gte: 1665637698}}).limit(1)

This doesn't cause scan.

However, if you are expecting more than one match, limit(2) or above will cause a full scan.

So to make it work for multiple matches, here is the second trick:

Add this index:

{start_address:-1, end_address:1}

Query with a $maxScan parameter:
db.ip_blocks.find({start_address:{$lte: 1665637698}, end_address:{$gte: 1665637698}})._addSpecial("$maxScan", 100)

As the addresses are ordered in a way helping us to scan, the correct records can be retrieved and also with limited scanning and thus it becomes fast.

How many bugs does Google have?

After I uploaded photos to my Google drive from my iphone today, they are all corrupted. Only a few were uploaded correctly. Google service degraded so much in the recent years. It really sucks now.

I am so glad I found out this early so I didn't lost many data.

How to estimate income level / ethnic ratio from your traffic

Use this API to get demographic data:

http://www.census.gov/data/developers/data-sets.html

For each visit to your site, convert ip to zip code by maxmind; use zip code to retireve income level / ethnic ratios by the census API. Sum the ratios by proportion - the result is the estimated income level / ethnic of your traffic (assuming people in a zip visit your site at random).

JavaScript supports functional programming!

I have been using JavaScript for quite long, believing it is an old language. Today I just went to across how to define my own map reduce function in Lua, and came into my thought: if I could do this in Lua, then it is doable in JavaScript... just to discover that JavaScript has supported functional programming for long!

Open your browser console with F12:

[1,2,3].map(function(a) {return a*a})     // shows [1, 4, 9]
[1,2,3].reduce(function(a,b) {return a+b})      // shows 6
[1,2,3].filter(function(a) {return a>2})      // shows [3]

Flatten does not come out of the box, it can be added by one line:

Array.prototype.flatten = function() { return this.reduce(function(a,b) { return a.concat(b)})}

[[0,1],[2,3],[4,5,6]].flatten()      // shows [0,1,2,3,4,5,6]


It made my day! :D

JavaScript is the best language invented ever!

ps: not supported by IE 8, which covers 8% of the Internet  users.. ouch!

Dealing with HHVM 3.1.0 memory leak

A week ago, I switched my site onto HHVM (3.1.0), in replace of PHP5. The throughput increase was significant, and response time dropped from 400ms to 300ms observed from Pingdom. I was very happy with the result.

After 5 days, I got a down-time on my site.. The reason being HHVM was shut down on itself. As it is serving through cgi port 9000. Nginx cannot access it. The solution was rather simple:

      sudo /etc/init.d/hhvm restart

However, I noticed my php-fpm setting was through socket rather than port. After some reading, cgi using socket is a lot more efficient than through port, as using port involving resolving through the network. So in order for nginx to read from hhvm socket, I opened up  /etc/nginx/hhvm.conf, replace the fastcgi_pass with:

    fastcgi_pass   unix:/var/run/hhvm/hhvm.sock;

I need to tell HHVM to serve from the socket, too. So, in /etc/hhvm/server.ini, I commented the server port and added file_socket:

    hhvm.server.file_socket=/var/run/hhvm/hhvm.sock
    ;hhvm.server.port = 9000

Restarting both nginx and hhvm, now it runs peacefully, again. no significant response time improvement, but hoping it fixes the issue of shutting down itself.

5 days after, the response time of site started to catch up.. :(




I ssh-ed into the node. Well, I found multiple processes of HHVM eating up the memory - that's not good. It is a memory leak. That's why it crashed a few days ago. and it only filled up the memory when serving through socket.

Keeping reading up HHVM, people had similar issues: Laravel + HHVM will cause memory leak: https://github.com/laravel/framework/issues/4757
It became my problem as my site is still on Laravel 3.

I guess I could switch back to PHP5. but I love the performance gained from HHVM and lambda expression, and would love to have it running around the bug.

I ran httperf against my site home page on localhost, and htop observing the hhvm processes, which consists a couple of mysql query and blade template render.  The memory grew fast with the requests. To rule out it is not the problem of db connection (or not just the problem of db): I ran httperf against another page that only have a static blade template like the following:

    public function action_about()
    {
        return View::make('home.about');
    }

And the memory consumption of hhvm grew just as fast. Third test with hello world: <?php echo 'hello' ?> it didn't cause hhvm any memory leak. 

Quite possibly it was caused by blade template engine running into some corner of hhvm. I didn't go deep - will leave the part for the future.

So I'm guessing hhvm did not limit on the max threads to be created, or the connections didn't timeout accumulated in the memory, or a background process should periodically expire the cache, or maybe there should need to be a memory cap defined, etc.. Have to try out and experiment with it. There wasn't much documentation, I could only found these information about server.ini settings:

https://github.com/facebook/hhvm/wiki/INI-Settings
        https://github.com/facebook/hhvm/blob/1cd8ceb06adf9a40039ebac40c489abb811ab599/hphp/runtime/base/runtime-option.cpp#L761

Tweaked more of the parameters, none of them stopped the memory accumulation! :( Here are the settings that I tried:

hvm.resource_limit.max_rss = 268435456
hhvm.resource_limit.drop_cache_cycle = 3600
hhvm.resource_limit.max_rsspolling_cycle = 3600
;hhvm.resource_limit.max_socket = 10
hhvm.resource_limit.socket_default_timeout = 30
hhvm.server.thread_count = 3
hhvm.pagelet_server.thread_drop_cache_timeout_seconds = 60


Finally, I found people talking about JIT is the cause of the memory leak, and the parameter trying to turn off JIT seems to work:

hhvm.jit = false

Now memory with hhvm processes stopped growing with httperf adding load to it. The service usable but I lost some performance - still beat PHP5. The problem is still unsolved. The remaining issue is: why does blade template engine cause JIT to memory leak?

PHP has lambda function! :D

I just discovered that you can perform lambda function with PHP tonight! Here's how to do it:

# First, go to php console  (you will need to install HipHop)

php -a


Welcome to HipHop Debugger!
Type "help" or "?" for a complete list of commands.

Note: no server specified, debugging local scripts only.
If you want to connect to a server, launch with "-h" or use:
  [m]achine [c]onnect <servername>

hphpd> 

# mapper function in PHP is called "array_map", it takes in a function and an array, just like Python:
$result = array_map( $a==> $a+1, [1,2,3,4])

$a ==> $a+1    is a lambda function, where $a is each element in the array. The function simply takes the value and plus one. the $result is expected to have [2,3,4,5]

$result will be another array of numbers.

To get an array of values from the $result variable, we need to do something similar to .to_a method in Ruby:

var_dump($result)

array(4) {
  [0]=>
  int(2)
  [1]=>
  int(3)
  [2]=>
  int(4)
  [3]=>
  int(5)

}

The result is as expected.

Power up my PHP site with Hip-hop!

Finally I made up my mind, and decide to try out Hiphop as a replacement for PHP5.

The installation was very straight-forward. I installed following the tutorial:

 http://fideloper.com/hhvm-nginx-laravel#comment-1325788095

In short:

$ sudo add-apt-repository -y ppa:mapnik/boost
$ wget -O - http://dl.hhvm.com/conf/hhvm.gpg.key | sudo apt-key add -
$ echo deb http://dl.hhvm.com/ubuntu precise main | sudo tee /etc/apt/sources.list.d/hhvm.list
$ sudo apt-get update
$ sudo apt-get install -y hhvm

$ sudo update-rc.d hhvm defaults

# Restart the service now
$ sudo service hhvm restart                      # We'll also restart HHVM

and inside my nginx config: /etc/nginx/sites-available/default, added one line "include hhvm.conf"

... Stuff above this omitted ...
    # Make site accessible from http://localhost/
    server_name localhost;
    include hhvm.conf;  # HERE'S THE MAGIC

    location / {
... Stuff below this omitted ...

Restarted now my site is running on Hip-hop! The performance improvement is significant. The requests per second increased about 5 times without any logic update! :D

(Some remaining issue: as my site is running on Laravel 3, I had to adjust some of the library to make it. Mostly 2 parts: Laravel 3 defined "yield" as a function for blade template engine, which is a reserved word in Hip-Hop. And Hip-hop came with its own Redis client, having a name clash with the Redis client from Laravel 3. If you ran into similar trouble, here is a diff patch to apply to make it compatible:

https://gist.github.com/yuhanz/afc6189330ce0c6120a8

If you have an old PHP site and are lazy about rewriting it, it is a good option to go with Hip-hop. They supports various PHP frameworks:

http://hhvm.com/frameworks/

Google search reached error code 500! :D

It is quite uncommon to see this. Might not be able to see it in another 100 years. So I made a screenshot! :D

WeChat vs. Google Hangout

I was chatting with my aunt from Los Angeles to Beijing over WeChat. It worked fines. The voice was sometime not very clear. The picture frame rate was low. But in general the service worked.

When switching to talk to her over Google Hangout, we experienced a very frequent drop of connections. Once the connection was established, I had to shut off video, in order to hear her voice at all.

It is quite interesting how the Chinese Company Tecent creates WeChat that enables the oversea video calls between nations, while Google service is only excelling local to America.

  

Tips on using Scala interactive mode

I enjoy interactive scripting tool like irb, but scala doesn't go very friendly as ruby. As I have to specify the class path in order to import my own packages. :/

So, I wrote a shell script automating it. As I always put all jar's in a lib/ folder, this script will get all the .jar files and add them into the classpath:

LIB_PATH=`find lib | tr "\n" :`                                                 

scala -cp .:$LIB_PATH

Save this into scala.sh, and run it at the directory where you have the lib/ folder. scala

Unfortunately, Java class names are not as simple and short as ruby. My memory is so bad that I cannot remember the package name of every single class. So, I create this shell script to help myself:

find lib -name \*.jar -exec unzip -v {} \; | grep -oh --color=never '[^[:space:]]*class$' | tr / . | sed -E 's/^(.*).class$/import \1/' > /tmp/imports.scala


Run this from where you lib/ folder is, and it will find all the classes with package names, and store into /tmp/imports.scala. Then in the interactive mode, you can search in this file and copy and paste the import class from a text editor. :D

It is always helpful to script common things in a text editor, such as initialize hbase connection, etc. Load them into scala by typing in the terminal so that we are not doing the boring thing again:
:load hbase_init_script.scala

The snippets are here on Gist:
https://gist.github.com/yuhanz/02155cfa921d6ef17014
https://gist.github.com/yuhanz/8002c696fbac9dd35ff2

Practice, Practice

More and more drawings

Almost exhausted the drawing practice book...

Well, my skill really improved after practising with Tadashi Ozawa's tutorial book. Drawing the figures from news paper and mangas never helped me that much. Thinking of creating original characters by my own now, maybe even some manga script.

Hot September

It has been really hot in these days. Plus power outage, which reminds me the California power crisis in 2003, when Arnold became our governor. Hopefully California will not be ripped off by out-of-state power companies, again.

Days without electricity gives me a feeling of living in North Korea, except in NOrth Korea the weather won't melt people down - but this is America; this is California, the greatest economical place on the earth. Isn't that weird? Or, ha, maybe the big earth quake is finally coming. God wants to give us some foresight... (wait, I am a budist, already. So, never mention God any more!)

Ok, here is the drawings that I have done during the Labor day weekend. Yeah, I had nothing better to do, except being half naked at home doing pencil drawing, and screaming out "This is Spartaaaaaa!".

More drawings