BC MACHINE LEARNING
Saved by @Mollyycolllinss

Okay, doing my first baby steps with r2frida (which combines the power of @radareorg and @fridadotre).

Gonna share my progress in this thread (live, so keep calm).

The goal: Runtime inspection of data sent out by TikTok !!before!! it gets encrypted

1/many

 First of all, we do not start from zero. I got some prior knowledge from past reversing attempts and want to share some important facts.

TikTok's (log data) encryption is accomplished by a native library. The Android Java code just serves as proxy function to the native function
The decompiled code for the respective native JNI function of an older TikTok version looks something like this, but in this example I use the most current TT version (no statical analysis done, yet)
In case you never reversed native libraries which were build to interface with Android Java layer via JNI, I highly suggest the entry level introduction on the topic by @maddiestone

https://t.co/T63vo3N4fw
Before we start, I want to pinpoint some important aspects (which are also covered by Maddie's videos).

1) Unlike raw C-functions, JNI functions like the one showcased above, receive pointers to complex Java objects .

F.e. a function receiving a String on the Java layer...
... would receive a pointer to a 'jstring' on the native layer (not a zero-terminated C-String).

In order to retrieve a C-String, to go on working with it in the native code, some translation functionality is required. This functionality is provided by the ...
JNI (Java Native Interface). The JNI environment is passed in to JNI functions as first parameter.

If you look at the example screenshot again, you see exactly this. Functions provided by the 'env' pointer are used to parse the Java function arguments (f.e. jByteArrays) ...
Once the raw data is converted to a more C-ish form, it gets passed to a inner function 'ss_encrypt' in my example. The inner function, in this case, is a pure C function and thus receives only C-style parameters (also no 'env' parameter, so it would not be able to access JNI)
A 2nd important aspect on JNI libraries, covered by @maddiestone

2) There are two ways to expose JNI methods from a native library:

a) export them with proper naming convention, so that JNI could recognize same on library load
b) use the JNI functionality 'registerNatives'...
... to register the JNI functions once the library gets loaded.

The second method of registering methods is wel suited for obfuscated code, as the methods neither have to follow naming convention, nor do they have to be exported.
As you might expect, TikTok uses the 'registerNative' approach. The screenshot below shows log output from a custom tool, which monitors JNI methods registered by instrumented Android apps (TikTok's encryption method in the example)
If you would decompile the Java part of the TikTok apk, the encryption functionality (on an older version) would look something like this:
The Java method 'm18421a' receives a Java 'byte[]' and a Java 'int' as parameters and returns a 'byte[]', again.

Internally, this data is forwarded to the native JNI method 'ttEncrypt'.
The important aspect about this, is that the native 'ttEncrypt' JNI method has to accept those exact parameter types and thus has to "register" with a proper method signature.

We already saw this signature in a previous screenshot
The last line from the screenshot above, shows 3 things the native code has to provide for each method, when calling register natives:

1) the call address of the native function implementation (0x7d70d1d5 in example)
2) The function name (ttEncrypt)
...
3) The method signature, which is '([BI)[B' in this case and translates to:

'(' start of parameters
'[B' byte[]
'I' int
')' end of parameters
'[B' byte[] (return value)
So we keep this in mind: Even if the native library does not export the encryption method, it has to store the 1) funtion address, 2) name and 3) signature in a data structure, in order to provide it to 'registerNatives' once the library gets loaded by JNI
We now know almost everything we need, in order to head over to r2frida, except some important facts on my test setup:

- the app is inspected on a physical device, running Android 9
- the device uses a !!32bit!! ARM application core
As I am new to r2frida, chances are high that things could be achieved in an easier ways.

Now to get started, I already have the latest @fridadotre server running on my USB connected android device and 'frida-ls-device' shows it being ready-for-action
So let's spawn a fresh TikTok instance with r2frida, using the following command
The command 'launch'es the process, on the attached frida 'usb' device with the device id '4c197256' and the process's package name, of course is 'com.zhiliaoapp.musically'

Instead of 'launch', two other options could be used:
- 'attach' (would attach to an already running process, given by name or PID)
- 'spawn' (like 'launch', but the process would not be resumed automatically after attaching)
So for the warm up, let us use the Frida functionality, which alllows enumerating loaded Java classes. This nicely combines with the r2 syntax (concatenation of single-letter commands, '~' for grep)

Important: commands targeting the r2frida plugin have to be prefixed with '\'
The r2frida command to list classes is '\ic' (note the backslash prefix). The unfiltered result would be a bit overwhelming ...
... so we grep for classes including the term "crypt"
The '\ic ' command lists the !Java! methods of the respective class
The signature of the static method 'EncryptorUtil.a' should look familiar to us (if you read the first tweets). It represents the Java layer of the encryption method and is called 'a' in this version
Note: The information above would be enough, to Intercept the method from the Java layer (f.e. with @fridadotre or Xposed), in order to inspect the call arguments (the byte[] parameter represents the plain data before encryption)
... but we are here for the native layer and to inspect data at runtime, right?

So lets search the whole address space for our native method name 'ttEncrypt'

Note: If you'd use r2's ascii search nothing would happen, you have to use the '\' prefix to search with r2frida
The search ends with two hits:
The screenshot below shows, that the attempt to print a hexdump from the address of the first hit fails with r2, while r2frida (backslash prefix) works.

Reason: The memory region was not populated when r2 was started (encryption library was loaded after process launch)
I solved this issue like this:
1) Quit r2
2) Open r2 with r2frida, again, but this time **attach** to the already running process

et voila ... the memory offset is mapped and dumpable with 'px' (without backslash prefix)
Note: The last step is not necessary for a data hexdump, as you could still use '\px', but it turned out to be useful when it comes to printing the disassembly of "late loaded" code regions. This is because I sometimes struggled with '\pd', but 'pd' worked (+ various r2 views)
Having a closer look at the first hit of out string search for 'ttEncrypt', we notice that it is directly followed by a C-string with our method signature.

So chances are high, that this data is part of the structure which gets handed in to 'registerNatives'
Reminder: in order to register the 'ttEncrypt' method to JNI, the 'registerNatives' method requires a structure containing
- method name (C-string)
- method signature (C-string)
- method pointer (native pointer)
So the next step would be to search the process memory space for cross references to the address of this method name string (0x8448b74c). As I haven't applied any auto analysis, I use a simple hex search for this (in my case the byte order of the address has to be reversed ...
... to account for the architecture endianess).

The result is promising: Only one hit, for a search across the whole address space:
Printing the first 12 bytes from this XREF offset, reveals 3 pointers again (reversed endianess):

- 0x8448b74c (expected, method name pointer)
- 0x8448b756 (ptr to signature string, yay)
- 0x8448b1d5 (likely pointer to JNI method implementation)
So the layout of the 3 pointers from above speaks a clear language. Very likely this is the data struct passed in to 'registerNatives' and thud 0x8448b1d5 points to the native implementation of 'ttEncrypt'
Sorry, before going on I have to insert a small excurse on adressing/instruction sets on arm 32 (specific to my test setup). Anyways it is crucial:

Arm 32 supports two instruction sets "ARM mode" (32bit) and "Thumb mode" (16bit) which could be used interchangebly
In order to distinguish if a function call target (branch) should be interpreted as ARM or THUMB, the least significant bit (LSB) of the function address is taken into account

For ARM the LSB is 0 (even address)
For THUMB the LSB is 1 (odd address)
The actual instruction ALWAYS resides on an odd address.

This means the function address 0x8448b1d5 homes code in THUMB mode (16bit), while the first instruction resides at 0x8448b1d4

(sorry if it gets a bit complicated, will be clear in a second)
So if we print the disassembly at the (assumed) 'ttEncrypt' address, things look a bit weird
... this is because instructions are interpreted in ARM mode (32bit).

Lets fix this:
... looks much better, still the first instruction is off-by-one 🤣

No seriously, as explained, on arm32 we have to disassemble at [THUMB mode address - 1] = 0x8448b1d4
Nice, this looks like a proper function stub (note how the callee stores reg values on the stack, before moving on).

Now to get a feeling on how often this function is called, lets use 'r2frida' power to trace it.

Important: The thumb address has to be used here!!!
The resulting output of the '\dt' command, which places the trace hook also indicates that the function address maps to an offset in 'https://t.co/fxhDnQke2o' ... let us call this a "nice confirmation"

Some actions in the TikTok app ... trace logs for ttEncrypt-calls arrive
... cigarette break ... stay tuned (if the app crashes meanwhile, I'll start from scratch)
Let's remove the trace hook for now, with '\dt-*'
Remember my screenshot of a decompiled 'ttEncrypt' function from an older TT version. We traced the corresponding functions.

Trying to runtime-parse the function parameters, which represent Java object instances would be insane (maybe impossible)
... luckily, at least the old implementation, internally called a method 'ss_encrypt' which received a c-style byte array pointer and an integer representing the length as first two parameters.
It would be way easier to runtime-inspect these
Lets take a closer look on the disassembly of our assumed 'ttEncrypt' function, by seeking to its offset with 's 0x8448b1d5' and switching to a more suitable r2 view with uppercase 'V' command (press 'p' till the view changes to disassembly)
The view from above allows scrolling through the functions code with cursor keys. Most important: calls to other code parts (branches) are printed bold and suffixed with [1], [2] ...

Hitting [alt+1] moves us straight to the marked branch offset:
The code above looks not like a legit inner function (we do not care for alignment and inspect the next branch).

Hitting 'u' returns us to the parent function, followed by [alt+2] which brings us into the 2nd branch
The 2nd branch at 0x84483aa4 looks better (proper function stub). We could easily drift back to the static analysis world, to find further evidence for it being the inner 'ss_encrypt' function. But hey, we are working with instrumentation, so let us just inspect the calls
Remember: While we disassembled at 0x84483aa4, the code is THUMB mode. Thus the proper tracing address would be 0x84483aa5 (LSB set to 1), unlike you like crashes (restarting here would not be funny, 'cause thanks to ASLR all function offsets would differ)
In contrast to our first tracing attempt, we use the beautiful command for formatted tracing, which allows us to print out function parameters for each call in a predefined format.

Command syntax:
The screenshot below shows how I placed my trace hook. The 'pppp' means that the first 4 function parameters should be printed as hex values (pointers) for each call.

Ultimately 2 calls get logged

More from Machine learning

Jan Giacomelli
Jan Giacomelli
@jangiacomelli
🤔 Python generators

What are they? How to use them?

#Python

🧵Let's find out 👇

1️⃣ Python generators are lazy iterators delivering the next value when their .next() is called.

They are created by using the yield keyword

next() can be called explicitly or implicitly inside for loop

They can be finite or infinite


2️⃣ yield - where a value is sent back to the caller, but the function doesn’t exit afterward as with the return statement

The state of function is remembered.

For example, the number is incremented and sent back from yield at the consecutive call next()


3️⃣ Generator stores only the current state of the function - it generates next element on next() call and forgets the previous one -> it saves memory

For example, we don't need to store 1mio elements in memory to do something with each element


4️⃣ When you call a generator function generator object is returned - it's not executed yet

It executes only when next() is called

For example, that's why Exception is raised only on the next() call
MACHINE LEARNING
Roger Grosse
Roger Grosse
@RogerGrosse
Important paper from Google on large batch optimization. They do impressively careful experiments measuring # iterations needed to achieve target validation error at various batch sizes. The main "surprise" is the lack of surprises. [thread]

https://t.co/7QIx5CFdfJ


The paper is a good example of lots of elements of good experimental design. They validate their metric by showing lots of variants give consistent results. They tune hyperparamters separately for each condition, check that optimum isn't at the endpoints, and measure sensitivity.

They have separate experiments where the hold fixed # iterations and # epochs, which (as they explain) measure very different things. They avoid confounds, such as batch norm's artificial dependence between batch size and regularization strength.

When the experiments are done carefully enough, the results are remarkably consistent between different datasets and architectures. Qualitatively, MNIST behaves just like ImageNet.

Importantly, they don't find any evidence for a "sharp/flat optima" effect whereby better optimization leads to worse final results. They have a good discussion of experimental artifacts/confounds in past papers where such effects were reported.
MACHINE LEARNING
Advertisement
Pratham Prasoon
Pratham Prasoon
@PrasoonPratham
Here's everything you need to know about the math for machine learning.

(+ free resources )
🧵👇

Before diving into the math, I suggest first having solid programming skills.

For example👇

(2 / 18)

In Python, these are the concepts which you must know:

- Object oriented programming in Python : Classes, Objects, Methods
- List slicing
- String formatting
- Dictionaries & Tuples
- Basic terminal commands
- Exception handling

(3 / 18)

If you want to learn those concepts for python, these courses are freecodecamp could be of help to you.

🔗Basics:youtube∙com/watch?v=rfscVS0vtbw
🔗Intermediate :youtube∙com/watch?v=HGOBQPFzWKo

(4 / 18)


👉You need to have really strong fundamentals in programming, because machine learning involves a lot of it.

It is 100% compulsory.

(5 / 18)
MACHINE LEARNING
Pratham Prasoon
Pratham Prasoon
@PrasoonPratham
There are several skills needed for learning machine learning that no one talks about.

Here are some of them.
(from what I've learned over the past two years)
🧵👇

This thread aims to introduce to you some of the skills often ignored for learning machine learning but are actually very important.

These skills will enable you to learn concepts quickly and more efficiently in this field.

(2 / 9)

1⃣ Reading

Probably one of the most underrated skills on this list.

In machine learning, you HAVE to read a lot of articles, papers, documentation, and whatnot.

It is mostly due to the theoretical nature of this field that reading is an important skill to have.

(3 / 9)

2⃣ Strong fundamentals in programming

Machine learning is just a lot of programming mixed with math and data.

Having clear programming fundamentals is crucial for this field.

I highly suggest you learn about these if you are using Python for machine learning.👇

(4 / 9)

- Object oriented programming in Python :Classes, Objects, Methods
- Lists & List functions
- Dunder Methods
- List comprehension
- List slicing
- String formatting
- List, Dictionaries & Tuples
- *args,**kwargs

(5 / 9)
MACHINE LEARNING
Santiago
Santiago
@svpino
An introduction to one of the the most basic structures used in machine learning: a tensor.

🧵👇

Tensors are the data structure used by machine learning systems, and getting to know them is an essential skill you should build early on.

A tensor is a container for numerical data. It is the way we store the information that we'll use within our system.

(2 / 16)

Three primary attributes define a tensor:

▫️ Its rank
▫️ Its shape
▫️ Its data type

(3 / 16)

The rank of a tensor refers to the tensor's number of axes.

Examples:

▫️ The rank of a matrix is 2 because it has two axes.
▫️ The rank of a vector is 1 because it has a single axis.

(4 / 16)

The shape of a tensor describes the number of dimensions along each axis.

Example:

▫️ A square matrix may have (3, 3) dimensions.
▫️ A tensor of rank 3 may have (2, 5, 7) dimensions.

(5 / 16)
MACHINE LEARNING , TENSOR BASICS

You May Also Like

Vibhu Vashisth \U0001f1ee\U0001f1f3
Vibhu Vashisth 🇮🇳...
@VIBHU_Tweet
MAHARAJA CHHATRASAL BUNDELA was the great Bundela King who revolted against the Mughal ruler Aurangzeb.
Chhatrasal Bundela was born in year 1649 CE in Maur Hill of Bundela region to Champat Rai and Lal Kunwar. He was the descendent of Rudra Pratap of Orchha.


His ancesters were the vassals of Mughal Emperor. He drew inspiration from his father and revolted against the alien rule of Mughals. He was just 12 yrs old when his parents died in fight against the Mughal Army of Aurangzeb in 1661.

When his beloved father attained martyrdom in the year 166, the young Chatrasal was present in the last battle of his father against the Mughal and from these valiant campaigns,..

..he gathered the courage to free Bundelkhand from Aurangzeb’s rule.
He joined the Mughals initially, and was recommended by the Rajput King Mirza Jai Singh to be recruited in the Mughal Army in order to support their Deccan Campaigns.


But serving the alien rule of Mughals which brought chaos and misery to Bundelkhand was unacceptable for Chatrasal Bundela and his father’s incessant struggle against the Mughals kept the fire of independent Bundelkhand alive in his heart.
ALL
Steve Stewart-Williams
Steve Stewart-Williams
@SteveStuWill
THREAD: 12 Things Everyone Should Know About IQ

1. IQ is one of the most heritable psychological traits – that is, individual differences in IQ are strongly associated with individual differences in genes (at least in fairly typical modern environments). https://t.co/3XxzW9bxLE


2. The heritability of IQ *increases* from childhood to adulthood. Meanwhile, the effect of the shared environment largely fades away. In other words, when it comes to IQ, nature becomes more important as we get older, nurture less. https://t.co/UqtS1lpw3n


3. IQ scores have been increasing for the last century or so, a phenomenon known as the Flynn effect. https://t.co/sCZvCst3hw (N ≈ 4 million)

(Note that the Flynn effect shows that IQ isn't 100% genetic; it doesn't show that it's 100% environmental.)


4. IQ predicts many important real world outcomes.

For example, though far from perfect, IQ is the single-best predictor of job performance we have – much better than Emotional Intelligence, the Big Five, Grit, etc. https://t.co/rKUgKDAAVx https://t.co/DWbVI8QSU3


5. Higher IQ is associated with a lower risk of death from most causes, including cardiovascular disease, respiratory disease, most forms of cancer, homicide, suicide, and accident. https://t.co/PJjGNyeQRA (N = 728,160)
LIFE
Advertisement
Ramblings of a Sikh
Ramblings of a Sikh
@RamblingSingh
1/ Jassa Singh Ahluwalia & his role in the history of Shri Harmandir Sahib (A Thread)


2/ Jassa Singh was born in 1720/21 at a village called Ahlu in the Lahore District of Punjab.


3/ Jassa Singh was educated in Delhi, in Persian, Arabic & more, under the care & supervision of Mata Sundari Ji.

Mata Sundari Gurdwara in Delhi.


4/ In 1733, Jassa Singh's mama, Thag Singh (ਤਾਗ ਸਿੰਘ), requested Mata Sundari Ji's permission to take Jassa Singh back to Punjab.


5/ At first Mata Ji refused but eventually gave permission for Jassa Singh to go with his mama, giving Jassa Singh a blessing and weapons, informing him, it was now the right time to take up arms.
HISTORY
Aditya Todmal
Aditya Todmal
@AdityaTodmal
Master Thread of all my threads!

Hello!! 👋

• I have curated some of the best tweets from the best traders we know of.

• Making one master thread and will keep posting all my threads under this.

• Go through this for super learning/value totally free of cost! 😃

1. 7 FREE OPTION TRADING COURSES FOR


2. THE ABSOLUTE BEST 15 SCANNERS EXPERTS ARE USING

Got these scanners from the following accounts:

1. @Pathik_Trader
2. @sanjufunda
3. @sanstocktrader
4. @SouravSenguptaI
5. @Rishikesh_ADX


3. 12 TRADING SETUPS which experts are using.

These setups I found from the following 4 accounts:

1. @Pathik_Trader
2. @sourabhsiso19
3. @ITRADE191
4.


4. Curated tweets on HOW TO SELL STRADDLES.

Everything covered in this thread.
1. Management
2. How to initiate
3. When to exit straddles
4. Examples
5. Videos on
OPTION SELLING , OPTIONSTRATEGIES , CATEGORY
Eric
Eric
@ericbahn
1/ Recently, I learned about the concept of ‘struggle porn’. This refers to the fetish of sacrificing nearly everything (health, relationships, time) for your startup. I fear that this label is creating more harm than good.

2/ @nateliason wrote a fabulous article about this topic, which first introduced me to this concept. Everyone should read it:

3/ @alexisohanian also recently spoke about struggle porn, though referring to it as ‘hustle porn.’ (as the co-founder of @hustlefundvc, obviously I take issue with this pejorative definition of ‘hustle’, but that’s another topic)

4/ Struggling in itself isn’t necessarily bad. Let’s be real here--if you’re going to start a startup, you will struggle. Founders should expect to work harder generally than employees. You will be challenged in many ways and also be super rewarded in others. This is normal.

5/ However, if you are over-indexing on hard work without clear purpose and degrading your mental health/relationships in the process--then this is very bad and we obviously should not celebrate this behavior.
STARTUPS