Thursday, August 17, 2023

Short explanation on PEFT: Parameter Efficient Fine Tuning

Many pretrained large language models are out there for us to use. However, they may not be accurate for our purpose. Thus, the model needs fine tuning. 

Since the model is large, the idea is to: make a copy of the existing model, and select a small percentage of trainable features to retrain. With the new copy of the model, train the new copy with your data.




Note that the library does not work with any random model that you created, as the parameter in LoraConfig task_type=TaskType.SEQ_2_SEQ_LM sets an expectation of the model.

LoRa applies the summation with the existing matrices with Low-Rank Matrices to adjust the weights, which is a trick to create a large matrix by adding small amount of parameters
(I explained it earlier in this post.  ) Since only a small percentage of the features are trainable, the training is relatively fast.


This video explains how the LoRA training works internally: 
https://www.coursera.org/learn/generative-ai-with-llms/lecture/NZOVw/peft-techniques-1-lora




Thursday, August 10, 2023

Pytorch: How to clear GPU memory

import gc

# del optimizer
# del model
gc.collect()
torch.cuda.empty_cache()

Quick Note: Training with Low-rank Matrices

When training a large matrix M with size WxH parameters is expensive, instead take the matrix into the multiplication of 2 smaller matrices. For example: matrix A is in size of (Wx3) and matrix B is in size of (3xH). And let A * B = M to give back a matrix of WxH dimensions. Since W * 3 + 3 * H < W *H, less amount of parameters are required.

This technique is mentioned in both of the following videos:

 https://www.coursera.org/learn/generative-ai-with-llms/lecture/NZOVw/peft-techniques-1-lora

https://youtu.be/exVPXVFPMDk?t=205

Wednesday, August 02, 2023

Details in Positional Encoding for Transformer

The Attention is all you need paper mentioned positional encoding without lacking some details. I am going to write my understanding at those details

The formula is the following:

PE(pos,2i) =sin(pos/100002i/dmodel)

PE(pos,2i+1) =cos(pos/100002i/dmodel) 

The paper mentioned that the i is the dimension index of and dmodel is dimension of the embedding. If so, given the last i = dmodel -1,  2i will be out of the bound. So, that is not the correct explanation.

2i and 2i+ 1 here suggest even and odd dimension indices. At the even dimension indices, apply sine function; at the odd dimension indices, apply cosine function. So i is ranged from [0, to dmodel/2) and for each i, it generates 2 dimensions.

Once having the PE (Positional Encoding) value for a position, by the diagram in page 3, it is added to the embedding of the input.

new_embedding[pos, 2i] = embedding[pos, 2i] + PE(pos, 2i) new_embedding[pos, 2i+1] = embedding[pos, 2i+1] + PE(pos, 2i+1)

The embedding variable here is the embedding for each word in a sentence, and pos is the position of the  sentence. (It is a sentence - not the whole dictionary.)

This part of the StatQuest video clearly explained how embedding is calculated.


Sunday, July 23, 2023

Key, Query, Value Matrices in Self Attention

The Attention is all you need paper mentioned about an attention function with construction of three matrices Q, K, V without much explanation. Fortunately, this Youtube tutorial on attention explained well (voiced in Chinese). Here is a note that I took from the video.

In self attention, there is only one input, as a list of tokens, each of which is a word expressed as a vector embedding. Call this input X of m elements. Each Xi is the embedding of the ith word. The task to guess the ith word to output, by looking at all words and the i-th word in the input.

Wk, Wq, Wv are the parameter matrices to be learned. Each of them multiplies X to get Q, K, V.

1. It needs to look at all words, which is the Wk matrix multiply X. This matrix is called key matrix as it looks at all keys (words). K = Wk * X

2. It needs to look at the ith word Xi, which is transformed by Q matrix, aka query matrix. qi = Wq * Xi

3. Take the result of K from step 1 and multiply the qi in step 2 and take a softmax. Call this result Ai = Softmax(K.transpose * qi)

4. The context vector at ith location Ai and multiply it with V. Call this result Ci = V * Ai. Since Ai came from step 3 with a softmax , Ci is essentially a weighted sum of V, based on the weight Ai.

5. Take Ci into a Softmax Classifier to get an output word.


Also, Self Attention is a special case of Attention. For self attention, qi is calculated by the ith word in the input Xi. For attention, qi is calculated by looking at the previous output of ith word ( For the very first position, <start> token is considered as the previous output.)


Sunday, October 09, 2022

How to run AR.js basic example

Git clone the project of https://github.com/AR-js-org/AR.js

The example has to run from an http server. Opening the example file in the directory won't work. Install nodejs http server

    npm install http-server -g

In your terminal, change directory to the root of AR.js, and run

    http-server

    The server is serving the html files in http://127.0.0.1:8080. Get to http://127.0.0.1:8080/three.js/examples/basic.html in your browser. Allow the page to access camera. The page will start recording you. On your phone, do a Google image search for "hiro marker".  Display the marker on your phone and place in front of the camera. The polygon animation will render on the marker.



Friday, July 22, 2022

AWS Java SDK DynamoDBv2 Scan

AWS Java SDK DynamoDBv2 (com.amazonaws.services.dynamodbv2.document.Table) has a terrible API at performing scan operation. Against common sense, the table.scan(scanSpec) returns a ItemCollection object, which requires the developer to call ItemCollection.iterator() in order to trigger an actual scan. If the ItemCollection.iterator() method is not triggered, the itemCollection.lastLowLevelResult field will be null.

This doesn't work, and will reach Null Pointer Exception:

itemCollection = table.scan(scanSpec) 

System.out.println(itemCollection.lastLowLevelResult.items.size)

This will work - calling of iterator method is required to populate the itemCollection.lastLowLevelResult field.

itemCollection = table.scan(scanSpec)

          List<Item> items = new ArrayList() 

CollectionUtils.addAll(items, itemCollection.iterator()) 

System.out.println(itemCollection.lastLowLevelResult.items.size)

Monday, July 11, 2022

XGBoost Parameter

This is a quick documentation of my understanding of the XGBoost parameters

  • max_depth: how deep can one tree grow

  • num_rounds : how many trees are in a prediction model

  • learning_rate: the weight between applying result (residual value) to the next tree

  • alpha: regularization term. (related to pruning trees)

  • lambda: regularization term. (related to pruning trees)

  • gamma: minimum loss reduction (related to limiting the depth of a tree)

  • Reference: 

Saturday, June 11, 2022

Java: wait & notify, await & signal, park & unpark

There are several way to stop a thread in Java, to get awaken later. Here are their usages:

wait & notify

Every object has a .wait() and .notify() method. These methods must be called in a synchronized block. 

When a .notify() happened before .wait(), it will not awake the thread.

await & signal 

With ReentrantLock, a condition object can be pulled from the lock, by `lock.newCondition()`. When the lock is locked, .await() These methods needs to be call when the lock is in lock state.

When a .signal() happened before .await(), it will not awake the thread.

LockSupport: park & unpark

Unlike wait & await, LockSupport.park() and LockSupport.unpark(t) and doesn't need to be in a locked / synchronized block. Since LockSupport is permit based, unpark assign a permit to a thread, which can be later used in park . So the order of park and unpark is not strict. Notice that permit doesn't have a counter - it can only be used in 1 park call.


Thursday, June 09, 2022

Java Locks: synchronized. ReentrantLock, ReentrantReadWriteLock, StampedLock

synchronized vs. ReentrantLock

Both create critical sections. ReentrantLock is unstructured and can lock and unlock in different methods. ReentrantLock can tryLock with a timeout.

ReentrantLock vs ReentrantReadWriteLock

ReentrantLock creates a critical sections that blocks both read & write. ReentrantReadWriteLock allows readLocks to read together, while blocking by critical sections when write is involved. Note that since readLock can block writeLock, it could result in writeLock starvation when a lot of readLocks appear. WriteLock cannot proceed until all readLocks are unlocked.

ReentrantReadWriteLock vs. StampedLock

ReentrantReadWriteLock is a pessimistic lock, which doesn't read while writer writes, and stops writer when it reads.

StampedLock is an optimistic lock, which reads (by tryOptimisticRead) while allowing write to happen, but also detects write (by validate) - if write happens during the time, simply reads again. 


Thursday, February 24, 2022

Getting Started with Haskell - Quick Tutorial

 To install Haskell compiler on MacOS:

brew install ghc

To compile a Haskell file into executable: (the Haskell file must have a main function)

ghc my-haskell-with-main-func.hs

To run without a Haskell file compiling: (the Haskell file will execute without a main function)

ghc -e ':script my-haskell-without-main-func-each-line-is-a-command.hs'

Haskell interactive console:

ghci

To output a string

putStrLn "hello"

To create a Hello World Haskell program

main = putStrLn "hello"

To create a multi-line Haskell program

  main = do

    putStrLn "hello"

    putStrLn "World"

To get stdin to a variable

t <- getLine 

To cast a String value into Int

i = read "123" :: Int

To covert a Int to String

s = show i

To round Float to Int

i = round  1.5

To split a String to a List by space

wordList = words "a b c"

To get item by index from a List 

wordList !! 2

To split a String to a List by new lines

 lineList = lines "hello world\nhi there"

To print a variable

print "hello"

To create a list of integer sequence from 1 to 5

[1..5]

To create a list of integers

[1,2,3] 

To define a lambda function

foo = \x -> x+1

To call a lambda function

foo 1 

To map a List  and apply a lambda function

map (\x -> x+1) [1 .. 2]

To filter a List  and apply a lambda function

filter (\x -> x>0) [1,2,3,0,5,6]

To take unique values of a list

import Data.List

nub [1,2,2,3,3,4]

To sort a List

sort [2,3,1] 

 To compose functions, use .

(map (\x -> x+1) . nub) [1,2,2,2,2]

To pipe functions from right to left without compose, use $

map (\x -> x+1) $ nub [1,2,2,2,2]

To flatten / merge multiple Lists into one List

concat [[1,2],[3,4]]

To join a List of String to one String with a delimiter

intercalate " " ["a", "b", "c"]  

To reduce, use foldr

foldr (\x s -> s+x) 100 [1,2,3]

foldr is kind of slow. Use foldl

foldl (\s v -> s+v) 100 [1,2,3] 

To sum numbers in a List

sum [1,2,3] 

To divide a List into a List of groups (Lists), each of which collects the repeated items in sequence:

groupBy (==) "aabbcccdddde" 

Write a function that returns constant value

f = (const 1)

f 100

To get combination of of a list with itself (Note: a String is a List)

        mapM (const "ABC") [1,1] 

To comment

-- This is a comment

To take power of a number

x ** 2     -- will result in Float value

x ^ 2       -- will result in Int value

To take absolute value

abs x

To take modulo of two values (% doesn't work)

mod 5 3 

To perform integer division

div 5 3 

To compare values with not equals (!= doesn't work)

1 /= x 

To get the max value in a List of values

maximum [1,2,3] 

To concat two Strings

"Hello" ++ "World"

To concat two Lists

[1,2,3] ++ [4,5,6]

To reverse a List

reverse [1,2,3] 

To take a for loop (

mapM_ (\i -> do {

    print $ "hello" ++ show i

})  [1.. 5]

-- Note: what do {} is used, remember to add ; at the end of each line, except for the last line.

-- Note: use mapM_ for side effect, and expect mapM_ to return no value.

To use ternary operator

x = 2

y = if x > 1 then "YES" else "NO" 

To define a variable in multi-line program

let { x = 2 }; 

To create a pair and get its left and right value, use a tuple:

 a = (1,2)

v1 = fst a

v2 = snd a

To zip two Lists into a List of pairs (tuples)

zip [1,2,3] ['a','b','c'] 

To define a lambda function that takes a tuple as input parameter:

(\(a,b) -> a+b) (1,2) 

Can I transpose a matrix? Yes, you can

import Data.List 

transpose [[1,2],[3,4]] 

Take length of a List

length [1,2,3] 

Take the first few from a List

take 1 [1,2,3]

Remove the first few from a List

drop 1 [1,2,3]

Drop the last one from a List

init [1,2,3,4,5]

Get the first value in a List

head [1,2,3] 

Get the last value in a List

last [1,2,3] 

Traverse a List until it doesn't match a condition

takeWhile (\x -> x > 0) [1,2,3,0,5,6] 

To get a String (List) of repeated values of a certain length

take 10 $ repeat '_' 

To define a recursive function: (you can define what a function return when a certain value is received at the parameter.)

f 0 = 0

f 1 = 0

f 2 = 1

f n = f (n-1) + f (n-2)

 To define a function f with multiple scenarios by conditions

f v

 | v > 0 = ">0"

 | v < 0 = "<0"

 | otherwise = "0"

To dynamic programming with recursion, see a tutorial on Data.Function.Memoize for fibonacci.

For example https://gist.github.com/yuhanz/e1c6793d3e8cb39fac0fa0ab9685235a


Thursday, February 10, 2022

Quick Tutorial: What is XML External Entities (XXE) Attack?

Vulnerable Scenario: your service takes in XML as input, and respond the content from the input (usually on error to indicate some parameter value).

Because XML has a DOCTYPE for variable replacement, you can easily define a variable to be replaced in the XML. For example, to define a variable myVar = "hello"

    <!DOCTYPE Query [ <!ENTITY myVar "hello" > ]>

This can be further extended to read a file on your disk for the content and assign it to the variable:

    <!DOCTYPE MySearchKeyword [ <!ENTITY myVar SYSTEM "file:///etc/passwd" > ]>

The attack: combine file reading with your XML input:

<?xml version='1.0' encoding='ISO-8859-1'?><!DOCTYPE Query [ <!ENTITY myVar SYSTEM "file:///etc/passwd" > ]> <Search>&myVar;</Search>

After our server will take the content inside <Search> to search (which is your passwd file), and it will respond with the file content to the client.


Solution:

 - Disable DTD feature in XML.

Wednesday, February 09, 2022

Quick Tutorial: What is Server-Side Request Forgery (SSRF)?

Vulnerable Scenario: when your app allows a user to send a URL to curl (or fetch, etc), potentially the user can curl a file on your server with:

curl file:///etc/passwd

So to improve the security against this, apply a check on the URL schema (not to accept with URL starting with file://)

Tuesday, January 11, 2022

FTL template - Cheatsheet

 This is a short documentation on how to use FTL template (FreeMarker Template Language)

https://freemarker.apache.org/


To try your Freemarker template online, use this interactive tool:

https://try.freemarker.apache.org/


How to check string not blank in FTL?

<#if myVariable.name?has_content>

How to null check in FTL?

<#if myVariable.name??>

How to import another FTL file?

<#import "/partials/my-partial-form.ftl" as partial>

How to call a macro in a partial form in FTL?

<@partial.myMacro myParam1 = 123/>

How to print timestamp as ISO string in FTL?

${myVariable.timestamp?datetime?string.iso}

${myVariable.timestamp?datetime?string.iso}

How to trim string in FTL?

${envelope.customer.lastName?trim}

How to keep 1 digital after decimal point in FTL?

#{y; m1}

Tuesday, December 07, 2021

How to "SSH" into an AWS EC2 Instance without SSH key

 This gives you terminal access to your EC2 instance. You will just need your AWS CLI setup, with permission to SSM. This is not really using "SSH". So the machine doesn't need SSH port open, nor need an SSH key to login.

1. Find out your EC2 instance id from AWS Console.

  Example instance id: i-0f85466ff323216f8

2. Run the SSM command:

 aws ssm start-session --target i-0f85466ff323216f8

  This should give you a terminal to the instance.

How to "SSH" into an AWS ECS Fargate Instance

This works for Fargate or Farget_spot to get terminal access, using SSM. This is not really using SSH. Your ECS instance doesn't have to have SSH port open for this to work.

1. Enable ExecuteCommand on your ECS service.

    aws ecs update-service --service myservice-v2 --enable-execute-command --cluster mycluster --region us-east-1

2. Update the Task Role of the myservice-v2 with: FullSSMPermission

    To find out the name of the task role, use AWS console and find it listed under your ECS service)

3. Start a new task in that service, and remember its task id. 

    Usually this can be done by stopping a task and let autoscaling policy to bring up a new node. The newly started task will have the ExecuteCommand setting that you setup earlier. This makes it possible to get to its terminal.

    The task id looks like something in this format: 01b46facf93d44b2ba3e3cf296dcaa30

4. "SSH" into the new node by its task id.

    aws ecs execute-command --region us-east-1 --cluster mycluster  --task 01b46facf93d44b2ba3e3cf296dcaa30 --container myContainer --command /bin/bash --interactive

    This should give you a terminal to the Fargate node.

5. If you cannot ssh in, find out what's missing in your setting from this checker:

    https://github.com/aws-containers/amazon-ecs-exec-checker

    Install the missing libraries until your machine passes the checker.

Wednesday, December 01, 2021

Getting Started with Clojure - Quick Tutorial

How to run a clojure file from terminal?

 clj /tmp/my.clj


How Clojure to read from stdin?

(print (read-line))


How to pass a file through stdin to a Clojure program from terminal?

cat /tmp/input.txt | clj /tmp/my.clj


How to hello world in Clojure?

(print "hello")

How to read all lines from stdin? (assuming less than 500 lines)

(print (take-while some? (repeatedly 500 #(read-line) )))

How to define a variable? (a variable that doesn't change.. ok! a constant! )

(def n (read-line))

How to parse string in stdin to an integer?

(def n (Integer/parseInt (read-line)))


How to map a lambda function on a list?

(map count ["hello", "world"])


(map #(count %) ["hello", "world"])


How to map a lambda function on a list, and do many things with each item?


(map #(do (print "hello") (count %))  ["hello", "world"])


How to print hello 3 times?


(map println (repeat 3 "hello"))



How to get an element in a list?


(get ["hello", "world"] 1)


How to zip two lists? - concat two lists


(map vector [1 2 3] [4 5 6])


How to create a hashmap for look up?


(def lookup (zipmap [1 2 3] [4 5 6]))


How to get a value from a hashmap by key? - same as getting an element in a list

(get lookup 3)



How to define a function?


(defn add [a b] (+ a b))




Friday, April 09, 2021

Apollo GraphQL Client: The order of Persisted Query Link & Retry Link Matters

We enabled automatic persisted query on an app that uses Apollo GraphQL client to query GraphQL. GraphQL persisted query is implemented by Apollo as an effort to save bandwidth from avoiding submitting the query both every time: https://www.apollographql.com/docs/apollo-server/performance/apq/ This was easily enabled with

    link = createPersistedQueryLink().concat(link)

Though at most of time the queries were submitted with persisted query hash, strangely there are times when full requests without query hash are still submitted, and the error cannot be recreated following the referer page of the unexpected request. It suggests the unexpected requests happened only occasionally.

I noticed that our link included the RetryLink feature.

         link = concat(new RetryLink({attempts: {max:2}}), new HttpLink({uri, fetchOptions}))

 Since Retry Link concatenated only with HttpLink, it will only know to send full request. Thus, the correct concatenated order is RetryLink on the outside of the PersistedQueryLink.

          link = concat(new RetryLink({attempts: {max:2}}), createPersistedQueryLink(), new HttpLink({uri, fetchOptions}))

After reordering the link concatenation, the client started to work correctly with persisted query hash all the time.

Tuesday, January 12, 2021

Java InheritableThreadLocal Does not Work Well with ThreadPool

    One of the dev's brought up the topic of InhertiableThreadLocal, which is a ThreadLocal whose value is copied from the parent thread to local thread. This first appeared to be an amazing feature. However, in the fine text, it says:

“Whenever that thread creates a new thread (a child thread), the InheritableThreadLocal object is automatically updated so that the new child thread has the same value associated with it as the parent thread”

    It means the copying process only happens during the creation. The Parent Thread and Child Thread relation relies on creation. This becomes an issue when you are using a ThreadPool, where the threads are reused instead being created. Some other people also encountered this issue:

So in conclusion, InheritableThreadLocal is not a good idea. Use caution when you are working with it.

How else would I pass ThreadLocal values across thread?

    You will want to copy the values in ThreadLocal variables as a field of a Runnable in the constructor. This creates a context for your Runnable. In your Runnable.run(), apply the values in the context to the ThreadLocal variables again. The allows ThreadLocal to be copied over because the constructor runs in your caller thread, while Runnable.run() runs in the invoked thread.

Thursday, July 02, 2020

Subtle Difference between NodeJS function and Arrow function.

There are 2 ways to define a function in NodeJS:

  • to define with the function keyword
  • to define with arrow "=>" keyword 

However, the functions defined by the two ways differ very subtly. Considering the .bind function on a function, which assigns an object as this of the function. the .bind function is available at functions created in both ways, but the function defined with arrow will not have access to this.

https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_objects/Function/bind

This snippet demonstrated that after calling .bind, the arrow create function still doesn't have access to the correct this object. (verified in NodeJS v13.10.1)