Designing Tiktok

Build a TikTok of your own. TikTok is a short-form, video-sharing app that allows users to create and share 15-second videos, on any topic.

Courtesy of Bytedance

Developing an app that can let users create & share videos could be a cakewalk for many seasoned developers of you, but designing the complete infra of an app with 100 million monthly active users is a different game.



Lets walk through a small story to understand the Logical Design :

X created a lovely Video Sharing App after months of hard work. His 2 friends liked it & started using it.

His friends uploaded daily 2 videos that gets stored on a 10GB Server Space that X bought.

Number of users start increasing once X's app is on TV, a sudden increase of 10,000 users. Server Space soon exhausted & the new users are unable to upload new videos.

X bought more server space, but that too soon exhausted with increasing users.

X realized he need to reduce video size before upload. He wrote a smart code that reduces video size without compromising on video quality. Now clients have to do too much work before upload so X moved this code to its Application Servers.

X bought new Application Servers with high processing powers for quick turnaround time. (Sponsors were attracted).

Another TV ad brought total users to be 1million, things started falling apart, Application Server couldn't reply all of the requests & finally it went down. Users were angry, Sponsors were unhappy.

X realized he needs Replicas of Application Servers such that when one Application Server dies another takes up its place & user has a notion that they are being served by single server . X brought 10 new Application Servers & a Load Balancer that evenly distributed load among the Application Server. Everyone is happy again.

With more Server Replicas in place , each video is being uploaded to all the Application Server which takes a lot of time, X wants the system to be quick & consistent. He decides the app can be eventually consistent by copying the video file to 2/3rd of servers and rest can keep uploading in background. This made the app availability high.

Each Server replica communicates to each other via Network, X designed a wonderful algorithm to make the system Network Failure Tolerant by integrating polling, retries.

As the system scales, more are the components required to handle the load. This story covers the Logical Designing of the app, majorly involving CAP Theorem.

Extracting all HBase columns

Have been fiddling with HBase? or new to HBase? and well-versed with our very favourite Hive. Here is a quick post to get all the columns in a HBase table.

Here is the usecase, Hbase is :

Apache HBase™ is the Hadoop database, a distributed, scalable, big data store. Apache HBase is an open-source, distributed, versioned, non-relational database

 Since its not a relational database so no concept of tables, rows, columns. Just data, that could be in any form of key value pair. So its like a JSON where each JSONObject may have unique data regardless of any fixed structure.

So here keys are the columns and their value is tuple for that JSONObject. In realtime scenarios a HBase table has a whole lot of JSONObjects and each object has whol lots of key-values. Now the hard part is how to find how many keys are there to make it structured or to convert it to Table format or any other business use-case.

1. Step1:

You have this HBase table hivehbase

On Hbase shell

create 'hivehbase', 'ratings'

put 'hivehbase', 'row1', 'ratings:userid', 'user1'

put 'hivehbase', 'row1', 'ratings:bookid', 'book1'

put 'hivehbase', 'row1', 'ratings:rating', '1'

 

put 'hivehbase', 'row2', 'ratings:userid', 'user2'

put 'hivehbase', 'row2', 'ratings:bookid', 'book1'

put 'hivehbase', 'row2', 'ratings:rating', '3'

 

put 'hivehbase', 'row3', 'ratings:userid', 'user2'

put 'hivehbase', 'row3', 'ratings:bookid', 'book2'

put 'hivehbase', 'row3', 'ratings:rating', '3'

 

put 'hivehbase', 'row4', 'ratings:userid', 'user2'

put 'hivehbase', 'row4', 'ratings:bookid', 'book4'

put 'hivehbase', 'row4', 'ratings:rating', '1'

 


Create column family by name 'ratings' and columns inside it

 

2.Now just get the column family names as below:

hbase(main):017:0> describe 'hivehbase'
Table hivehbase is ENABLED                                                                                                                  
hivehbase                                                                                                                                   
COLUMN FAMILIES DESCRIPTION                                                                                                                 
{NAME => 'ratings', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MI
N_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}       

Here column family name is 'ratings'

 

 

3.Follow https://cwiki.apache.org/confluence/display/Hive/HBaseIntegration to integrate HBase to Hive and add Hbase-Hive Storage Handler to Hive libs

On the Hive shell 

hive>add jar /usr/lib/hbase/lib/hbase-common.jar;
add jar /usr/lib/hbase/lib/hbase-client.jar;
add jar /usr/lib/hbase/lib/zookeeper.jar;
add jar /usr/lib/hbase/lib/hbase-common-0.98.0.2.1.1.0-385-hadoop2-tests.jar;
add jar /usr/lib/hbase/lib/guava-12.0.1.jar;

hive> CREATE EXTERNAL TABLE hbase_table_hive(peData map, row_key int)
STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
WITH SERDEPROPERTIES ("hbase.columns.mapping" = "ratings:,:key")
TBLPROPERTIES ("hbase.table.name" = "hivehbase");

 

 

4.Now your HBase table is also visible on Hive as  a table. To view all the columns of this table simply perform a select:

hive>select * from hbase_table_hive;
OK
{"bookid":"book1","rating":"1","userid":"user1"}    NULL
{"bookid":"book1","rating":"3","userid":"user2"}    NULL
{"bookid":"book2","rating":"3","userid":"user2"}    NULL
{"bookid":"book4","rating":"1","userid":"user2"}    NULL

It will list all the keys in the system in column family 'ratings' . Similarly you can get all the columns of all column families. If there is no value for a column it will show key and value is null. 

Also you can convert this map JSON list of columns to separate column in another Hive step.

/dev/random/2

yes my blog has a new address now dotSlashA.in 
yes you guessed it right! & in-case u dint guessed here is what ./a.in is
"an antonym to ./a.out"
which again expresses my love for open-source.
Oh c'mon ./a.out is ... defined at Wikipedia.

So nowadays spending less time on SO & busy in Moving all my stuffs to ./a.in.

Though I believe ./a.in should be a collaborative site to post rammblings but.. may be later Or.. yes mail it to admin@dotSlashA.in to express interest to have a username created for you start contributing.

& yeah Annual Stack Overflow Meetup Day! in Indore!! Count me in.

./Saurabh

/dev/random

So Again its really long time since my last post.


Amazing this time is  this guy whose blog forced me to write this post.Okay enough praising.


Did we talked about commandlineflu?Nope ! I think I forgot .It's been one of the longest pending post idea .So start tricking other Users by 

• Shutdown a Windows machine from Linux

• Save a file you edited in vim without the needed permissions

• Copy ssh keys to user@host to enable password-less ssh logins.

• start a tunnel from some machine's port 80 to your local post 2001

• Execute a command without saving it in the history

• or any kind of linux

Here ends my random post generator.
bash: ./saurabh: command not found

The Fun Part

So it isn't new but .. sounds great 
so lets go ahead ..

Here’s where it gets fun: many of these devices use hard-coded SSL keys that are baked into the firmware. That means that if Alice and Bob are both using the same router with the same firmware version, then both of their routers have the same SSL keys. All Eve needs to do in order to decrypt their traffic is to download the firmware from the vendor’s Web site and extract the SSL private key from the firmware image.

Running Android Native Code in your Android app

Just came over a post asking for Android's Logging Best Practices that took me to android log collector. Delving more deeper into log collector's code I got the below code to executes the specified command and its arguments in a separate native process.

All you need is just below code

String myStringArray[]= {"logcat","-d"};

Process process = Runtime.getRuntime().exec(myStringArray);

Setup the permissions for the required operation & you r done!Enjoy

But keep in mind Use Native Methods Judiciously & do check jon oberheide's Android Hax
And thanks to android-log-colllector team for the code.

Though the post describes a way of dropping a shell command from an app , check this for more ways to run shell commands from your app.
And also this  & I have heard MarketEnabler source offers a nice ShellInterface class.
So go get your app runnning!

See ya there at ./a.in

Stacks OverFlowing..

See ya there at ./a.in