How to "SSH" into an AWS EC2 Instance without SSH key

 This gives you terminal access to your EC2 instance. You will just need your AWS CLI setup, with permission to SSM. This is not really using "SSH". So the machine doesn't need SSH port open, nor need an SSH key to login.

1. Find out your EC2 instance id from AWS Console.

  Example instance id: i-0f85466ff323216f8

2. Run the SSM command:

 aws ssm start-session --target i-0f85466ff323216f8

  This should give you a terminal to the instance.

How to "SSH" into an AWS ECS Fargate Instance

This works for Fargate or Farget_spot to get terminal access, using SSM. This is not really using SSH. Your ECS instance doesn't have to have SSH port open for this to work.

1. Enable ExecuteCommand on your ECS service.

    aws ecs update-service --service myservice-v2 --enable-execute-command --cluster mycluster --region us-east-1

2. Update the Task Role of the myservice-v2 with: FullSSMPermission

    To find out the name of the task role, use AWS console and find it listed under your ECS service)

3. Start a new task in that service, and remember its task id. 

    Usually this can be done by stopping a task and let autoscaling policy to bring up a new node. The newly started task will have the ExecuteCommand setting that you setup earlier. This makes it possible to get to its terminal.

    The task id looks like something in this format: 01b46facf93d44b2ba3e3cf296dcaa30

4. "SSH" into the new node by its task id.

    aws ecs execute-command --region us-east-1 --cluster mycluster  --task 01b46facf93d44b2ba3e3cf296dcaa30 --container myContainer --command /bin/bash --interactive

    This should give you a terminal to the Fargate node.

5. If you cannot ssh in, find out what's missing in your setting from this checker:

    https://github.com/aws-containers/amazon-ecs-exec-checker

    Install the missing libraries until your machine passes the checker.

Getting Started with Clojure - Quick Tutorial

How to run a clojure file from terminal?

 clj /tmp/my.clj

How Clojure to read from stdin?

(print (read-line))

How to pass a file through stdin to a Clojure program from terminal?

cat /tmp/input.txt | clj /tmp/my.clj

How to hello world in Clojure?

(print "hello")

How to read all lines from stdin? (assuming less than 500 lines)

(print (take-while some? (repeatedly 500 #(read-line) )))

How to define a variable? (a variable that doesn't change.. ok! a constant! )

(def n (read-line))

How to parse string in stdin to an integer?

(def n (Integer/parseInt (read-line)))

How to map a lambda function on a list?

(map count ["hello", "world"])

(map #(count %) ["hello", "world"])

How to map a lambda function on a list, and do many things with each item?

(map #(do (print "hello") (count %))  ["hello", "world"])

How to print hello 3 times?

(map println (repeat 3 "hello"))

How to get an element in a list?

(get ["hello", "world"] 1)

How to zip two lists? - concat two lists

(map vector [1 2 3] [4 5 6])

How to create a hashmap for look up?

(def lookup (zipmap [1 2 3] [4 5 6]))

How to get a value from a hashmap by key? - same as getting an element in a list

(get lookup 3)

How to define a function?

(defn add [a b] (+ a b))

Apollo GraphQL Client: The order of Persisted Query Link & Retry Link Matters

We enabled automatic persisted query on an app that uses Apollo GraphQL client to query GraphQL. GraphQL persisted query is implemented by Apollo as an effort to save bandwidth from avoiding submitting the query both every time: https://www.apollographql.com/docs/apollo-server/performance/apq/ This was easily enabled with

    link = createPersistedQueryLink().concat(link)

Though at most of time the queries were submitted with persisted query hash, strangely there are times when full requests without query hash are still submitted, and the error cannot be recreated following the referer page of the unexpected request. It suggests the unexpected requests happened only occasionally.

I noticed that our link included the RetryLink feature.

         link = concat(new RetryLink({attempts: {max:2}}), new HttpLink({uri, fetchOptions}))

 Since Retry Link concatenated only with HttpLink, it will only know to send full request. Thus, the correct concatenated order is RetryLink on the outside of the PersistedQueryLink.

          link = concat(new RetryLink({attempts: {max:2}}), createPersistedQueryLink(), new HttpLink({uri, fetchOptions}))

After reordering the link concatenation, the client started to work correctly with persisted query hash all the time.

Java InheritableThreadLocal Does not Work Well with ThreadPool

    One of the dev's brought up the topic of InhertiableThreadLocal, which is a ThreadLocal whose value is copied from the parent thread to local thread. This first appeared to be an amazing feature. However, in the fine text, it says:

“Whenever that thread creates a new thread (a child thread), the InheritableThreadLocal object is automatically updated so that the new child thread has the same value associated with it as the parent thread”

• https://www.oreilly.com/library/view/java-in-a/0596007736/re145.html

    It means the copying process only happens during the creation. The Parent Thread and Child Thread relation relies on creation. This becomes an issue when you are using a ThreadPool, where the threads are reused instead being created. Some other people also encountered this issue:

• https://stackoverflow.com/questions/9012371/using-inheritablethreadlocal-with-threadpoolexecutor-or-a-threadpoolexecut

So in conclusion, InheritableThreadLocal is not a good idea. Use caution when you are working with it.

How else would I pass ThreadLocal values across thread?

    You will want to copy the values in ThreadLocal variables as a field of a Runnable in the constructor. This creates a context for your Runnable. In your Runnable.run(), apply the values in the context to the ThreadLocal variables again. The allows ThreadLocal to be copied over because the constructor runs in your caller thread, while Runnable.run() runs in the invoked thread.

Subtle Difference between NodeJS function and Arrow function.

There are 2 ways to define a function in NodeJS:

• to define with the function keyword
• to define with arrow "=>" keyword 

However, the functions defined by the two ways differ very subtly. Considering the .bind function on a function, which assigns an object as this of the function. the .bind function is available at functions created in both ways, but the function defined with arrow will not have access to this.

https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_objects/Function/bind

This snippet demonstrated that after calling .bind, the arrow create function still doesn't have access to the correct this object. (verified in NodeJS v13.10.1)

Java 8: Passing a Method as an Argument with & without Side Effects

Pure function in Computer Science is defined as a method without side effect. Given the same input to a pure function, the developer can expect the same output. This makes the code simpler. This is typically implemented as static function in Java 8.

In contrast, when a method may have side effects, which could be due to internal states stored in an object or I/O actions, the outcome of the method is hard to predict. Thus harder to test.

To pass a function as an argument to a method in Java 8, you may perform on a static method or an instance method: https://docs.oracle.com/javase/tutorial/java/javaOO/methodreferences.html

In this example, we have a static method passed without side effect, while passing an instance method can potentially introduce side effects unaware to the programmer.

The output from the above code shows that states may be allowed to exist in an argument passed with an instance method:

/Library/Java/JavaVirtualMachines/jdk1.8.0_51.jdk/Contents/Home/bin/java ...
3
14
15

Process finished with exit code 0

An instance method can be a pure function, too. However, passing an instance method as an argument opens up for later developers to introduce side effects in the function.

Stay with static method when trying to make a pure function, and use instance method only when you intend to introduce states to a function.

Steps to Install Python Control library (For LQR control) On Raspberry Pi

I got a Raspberry Pi 3 and would like to program python on it with LQR for stabilized control. Here are the steps that I took to install the Python control package and its dependencies:

pip3 install control --upgrade
sudo apt-get install build-essential
sudo apt-get install ninja-build
sudo apt install gfortran
pip3 install slycot

To test the installed control package, run the following snippet:

>>> import control
>>> control.lqr([[1]], [[1]],[[1]],[[1]])
(array([[2.41421356]]), array([[2.41421356]]), array([-1.4142135+0.j], dtype=complex64))

Run AWS Lambda Function Code Locally - Quick Tutorial

Why do I want to Run the Lambda Function Code locally?

One of my dev friends falsely believed that AWS Lambda Function would require deployment in order to run. This is not true. It is a common misunderstanding since in the Lambda Function you are only defining a handler.

The solution is simple - just write a main function that includes your lambda handler and pass parameters to it! No deployment needed.

How to Do it?

Here's your lambda function in index.js

exports.handler = (event, context, callback) => {
  console.log('hello', event.name, 'from', process.env.SERVICE_NAME);
  callback(undefined, 'done');
};

Write your main.js

process.env.SERVICE_NAME = 'commandline lambda function';

const index = require('./index.js');

const context = {};

const event = {
  "name": "world"
}

index.handler(event, context, (err, data) => {
  if (err) { console.error('error: ', err); }
  if (data) { console.log('data: ', data); }
});

You have environment variables defined, have your event defined, and invoked the lambda function handler. Now just call it from the terminal:

node main.js

Note that it will use the credential in your ~/.aws/credentials file, just like the aws-cli tool.

How to run the command line lambda function with aws-okta credentials:

aws-okta exec my-aws-account-name -- node main.js

Getting Started with Vim - Quick Tutorial

Why would you want to use Vim?

When you ssh into a server, you will need a text editor that works in the terminal to change for example, server configurations. Nano is a text editor that works on terminal. But it is hard to navigate fast with Nano. Improve your Vim skill for the speed!

How to use Vim?

To open a file:

• In the terminal, type in:  vi /tmp/1.txt

To navigate:

• Your cursor keys just work
• Press 'control' + 'D' for pageDown
• Press 'control' + 'U' for pageUp

To get to the bottom of the doc:

• Press 'shift' + 'G'

To get to the top of the doc:

• Type in ':0' then 'enter'

To get to the start of the line:

• Press 'shift' + '^'

To get to the end of the line:

• Press 'shift' + '$'

To get to the 50th line of the doc:

• Type in ':50' then 'enter'

To delete the current line:

• Type in 'dd'

To delete the next 20 lines:

• Type in '20dd'

To delete the all lines:

• Type in '100000dd'

• Repeat until there's no line left

To save file and exit:

• Type in ':wq!' then 'enter'

To exit without saving:

• Type in ':q!' then 'enter'

To edit:

• Press the 'i' key to enter editing mode

• Your 'delete' key and 'backspace' key just work

To exit editing mode:

• Press the 'esc' key

To undo a change:

• Press the 'esc' key to exit the edit mode

• Press the 'u' key to undo

To get to a word:

• Press the '/' key to search forward

• Example: '/hello' will search forward the next hello that appears in the text

• Press the 'n' key to get to the next 'hello'

To get to a word from the bottom of the doc:

• Press 'shift' + 'G' to get to the bottom

• Press 'shift' + '?' key to search backward

• Example: '?hello' will search last hello that appears in the text

• Press the 'n' key to get to the next 'hello'

To delete a word:

• Use the above method to get to the word

• Press 'dw' to delete the word

To delete one character:

• Use the above method to get to the word

• Press 'x' to delete a character

To copy text: 

• Your cursor just works in selecting text
• Your Windows command just works: 'ctrl' + 'c'
• 'cmd' + 'c' for OSX

To paste text:

• Press the 'i' key to enter editing mode
• Your Windows command just works: 'ctrl' + 'v'
• 'cmd' + 'v' for OSX

 If your terminal does not support text selection with copy & paste, try:

To copy 1 line:

• Press 'yy' to copy the current line

To paste 1 line:

• Press 'p' to paste a copied a line

Great. You know how to use vim.

Delay Processing of a List Using JavaScript Promise without Recursion

In many cases, we'd like to perform a list of actions with delay in between. (Ex: to make a list of API calls without overloading the server.) However, JavaScript has no sleep function, but it is provided with a setTimeout method callback after a delay. Typically tail recursion can be applied at the end of function to trigger another setTimeout for the next execution. However this makes code ugly and can be avoided by using Promise. The setTimeout function can be easily wrapped into a promise to provide delay by:

function promiseDelay(miliseconds) { return new Promise((resolve) => { setTimeout(resolve, miliseconds)})}

To use this function to delay your call, just place the logic inside .then() method

promiseDelay(1000).then(() => {console.log('hello')})

When there's a list of values to be processed, we can keep chaining new deplayPromise at the end of the promise:

p = Promise.resolve(undefined)
[1,2,3].forEach(x=> {
  p = p.then(() => promiseDelay(1000).then(() => {console.log(x)}) )
})

This is a little hard to read, as there is .then() inside a .then(). Note that the next execution is not depending on the result of the delay, this is equivalent to wrap the execution into another promise, and use Promise.all() to ensure both delay and execution to be both completed before proceeding:

function promiseDisplay(x) {
  return new Promise((resolve) => { console.log(x); resolve()})
}

Promise.all([promiseDelay(1000), promiseDisplay(x)]);

Applying the above change, the code now looks nice:

Dealing with Connection Timeout to Redis when the "Reclaimed Items" Spike

Recently I encountered a Redis problem, where new Redis connections failed to be established from the server, and the client massively timed out. The AWS managed Redis cluster has a few metrics coincide with the surge of connection issue: "New Connections" and "Reclaimed Items" both spiked during the time of difficulty.

Commonly Redis would respond slow, causing connection issue on the client when "Eviction" events occurs. This usually happens where the max capacity of the Redis node has been reached, and entries are forced out of Redis as a result. However, in my case the "Eviction" metrics showed 0, and "Bytes Used for Cache" suggested that Redis has enough RAM. Clearly, the surge of "New Connections" is a symptom rather than a direct cause. When the system was halting, normally connections would be expected to build up waiting on a certain resource.

The AWS documentation shows that the "Reclaimed Item" is the
"The total number of key expiration event". https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/CacheMetrics.Redis.html

Redis documentation explains that "basically [key] expired events are generated when the Redis server deletes the key" after TTL on a key goes to zero. https://redis.io/topics/notifications Since the background Redis process may keep deleting items when the collected sample has >25% of expired item, Redis may take a few seconds to delete when large amount of items are expiring.  https://redis.io/commands/expire This would make established connections to be unable to proceed their operations, causing connection issue for the client.

Note that this problem is related to both write/second and the TTL length. You have an option of either reducing write/second, or make TTL shorter so that the number of items don't build up at a potential of being deleted all at once by the background process.

When will Hystrix Throw ThreadPoolRejection Error when Synchronous Queue is used?

Hystrix uses SynchronousQueue when MaxQueueSize has been set to -1, and uses LinkedBlockingQueue when MaxQueueSize is set to >=0.

When LinkedBlockingQueue is used and the number of concurrent requests is beyond (threadPool.CoreSize + queueSizeRejectionThreshold), ThreadPoolRejection Exception will be thrown.

When SynchronousQueue is used and the number of concurrent requests is beyond (threadPool.CoreSize), ThreadPoolRejection Exception will be thrown.

Here's a snippet providing it when SynchronousQueue is used:

submitting:0
submitting:1
submitting:2
submitting:3
submitting:4
submitting:5
running: 1
running: 2
running: 0
running: 4
running: 3
submitting:6
get Object of 6
Exception in thread "main" com.netflix.hystrix.exception.HystrixRuntimeException: TestHystrix$1 could not be queued for execution and no fallback available.
 at com.netflix.hystrix.AbstractCommand$21.call(AbstractCommand.java:783)
 at com.netflix.hystrix.AbstractCommand$21.call(AbstractCommand.java:768)
 at rx.internal.operators.OperatorOnErrorResumeNextViaFunction$1.onError(OperatorOnErrorResumeNextViaFunction.java:99)
 at rx.internal.operators.OperatorDoOnEach$1.onError(OperatorDoOnEach.java:71)
 at rx.internal.operators.OperatorDoOnEach$1.onError(OperatorDoOnEach.java:71)
...

Note that ThreadPoolRejection starts to happen when all 5 threads in the threadPool have been taken.

Vim Tip - Delete All lines above / below

vim command to delete all lines above - dgg
vim command to delete all lines below - dG

UTF8 Encoding Problem in Laravel Blade Template

I have a Chinese site written in three years ago. It is still on Laravel 3, but I'd like to migrate it into a new server. After the migration, it appears that all Chinese characters from the application are rendered twisted. And it was not because of the <meta> tag  that can fix it, as I am seeing the wrong character in the source.

My first step was try to render some Chinese characters from plain .php file. It worked. That ruled out the Nginx/HHVM configurations. After that, I started to place "print_r()" with some UTF-8 characters. Tracing to /laravel/core.php, it appears that mb_output_handler method is using a wrong default encoding. I made the following changes:

mb_http_output("UTF-8");    // to make blade template output in UTF-8
ob_start('mb_output_handler');

The UTF-8 characters appeared normal again after the change. Great! :)

List of Python Unicode / UTF8 String Encoding Snippet

\u to \x conversion

>>> print(u'\u0420\u0443\u0441\u0441\u043a\u0438\u0439')
Русский
>>> a = u'\u0420\u0443\u0441\u0441\u043a\u0438\u0439'.encode('utf8')
>>> a
'\xd0\xa0\xd1\x83\xd1\x81\xd1\x81\xd0\xba\xd0\xb8\xd0\xb9'
>>> print(a)
Русский


\x to \u conversion

Firstly, convert unicode to string (iso-8859-1 encoding)
Then you need to know the encoding. In this example, the encoding of the string is gb2312:

>>> x
u'\xcc\xd8\xbe\xaf\xc1\xa6\xc1\xbf'
>>> x.encode('iso-8859-1')
'\xcc\xd8\xbe\xaf\xc1\xa6\xc1\xbf'
>>> '\xcc\xd8\xbe\xaf\xc1\xa6\xc1\xbf'.decode('gb2312')
u'\u7279\u8b66\u529b\u91cf'
>>> print(u'\u7279\u8b66\u529b\u91cf')
特警力量

How to Make Slow Mongo Query on Compound Index with $gte and $lte Fast

I have a mongo collection of IpRanges, with start_address and end_address. I'd like to query given an IP. with compound index on {start_address:1, end_address:1}, this however is still very slow:

db.ip_blocks.find({start_address: {$lte: 1665637698}, end_address: {$gte: 1665637698}})


with .explain(), it appeared that this query causes a lot of scanning and slow, as only the $lte query worked hard, and the second query is scanning what's remaining from the first query.

Here is a trick to speed this up if you are only trying to get the first match:

db.ip_blocks.find({start_address: {$lte: 1665637698}, end_address: {$gte: 1665637698}}).limit(1)

This doesn't cause scan.

However, if you are expecting more than one match, limit(2) or above will cause a full scan.

So to make it work for multiple matches, here is the second trick:

Add this index:

{start_address:-1, end_address:1}

Query with a $maxScan parameter:
db.ip_blocks.find({start_address:{$lte: 1665637698}, end_address:{$gte: 1665637698}})._addSpecial("$maxScan", 100)

As the addresses are ordered in a way helping us to scan, the correct records can be retrieved and also with limited scanning and thus it becomes fast.

How many bugs does Google have?

After I uploaded photos to my Google drive from my iphone today, they are all corrupted. Only a few were uploaded correctly. Google service degraded so much in the recent years. It really sucks now.

I am so glad I found out this early so I didn't lost many data.

How to estimate income level / ethnic ratio from your traffic

Use this API to get demographic data:

http://www.census.gov/data/developers/data-sets.html

For each visit to your site, convert ip to zip code by maxmind; use zip code to retireve income level / ethnic ratios by the census API. Sum the ratios by proportion - the result is the estimated income level / ethnic of your traffic (assuming people in a zip visit your site at random).

JavaScript supports functional programming!

I have been using JavaScript for quite long, believing it is an old language. Today I just went to across how to define my own map reduce function in Lua, and came into my thought: if I could do this in Lua, then it is doable in JavaScript... just to discover that JavaScript has supported functional programming for long!

Open your browser console with F12:

[1,2,3].map(function(a) {return a*a})     // shows [1, 4, 9]
[1,2,3].reduce(function(a,b) {return a+b})      // shows 6
[1,2,3].filter(function(a) {return a>2})      // shows [3]

Flatten does not come out of the box, it can be added by one line:

Array.prototype.flatten = function() { return this.reduce(function(a,b) { return a.concat(b)})}

[[0,1],[2,3],[4,5,6]].flatten()      // shows [0,1,2,3,4,5,6]


It made my day! :D

JavaScript is the best language invented ever!

ps: not supported by IE 8, which covers 8% of the Internet  users.. ouch!