BC TECH
Saved by @SteveeRogerr

A trend I'm excited for this year: DataOps & the Analytical Engineer

~10 years ago DevOps was born. The role of system admins and developers merged. Infrastructure became self-serve

Today the role of data engineers and business analysts are merging. Data is becoming self-serve

 Data infrastructure is becoming so powerful that the tools today allow non-technical folks to carry out the once complicated / custom code/ huge backlog jobs of data engineers.

Before getting into what this means, let's first discuss how we got here
Before 2012 the data world was dominated by transactional (OLTP) databases like PostgreSQL, MySQL, etc and analytical (OLAP) databases like Oracle, Netezza

Tools like Informatica / Talend were used to batch load (ETL) data into these databases, Tableau used to visualize
As you can imagine, there was heavy engineering work to manage the environment...

Then in 2013 AWS released their cloud data warehouse Redshift, and it was a game changer. Snowflake was founded in 2012, but didn't really pick up steam until a few years later (around 2016)
So why was Redshift a big deal?

1. It was the first cloud-native OLAP warehouse. It reduced the TCO of an OLAP database by orders of magnitude.

2. Speed of processing analytical queries increased dramatically
3. And later on (Snowflake pioneered this) they separated compute & storage. In overly simplified terms, this meant customers could scale their storage and compute resources independently of one another. This was a huge deal

What does this all mean? An EXPLOSION of data
The barriers to maintain a database were completely broken down, and the amount of data that was sent to Redshift / Snowflake / BigQuery skyrocketed.

Now, after this point we still weren't ready for DataOps / Analytical Engineers. What did it take to get there?
In my opinion there were 3 major technologies / shifts that happened that have given rise to DataOps:
1. The shift from ETL to ELT (extract-transform-load to extract-load-transform). Data used to be transformed (joined, aggregated, cleaned, etc) in motion while being loaded into the warehouse. Now, data is being loaded into the warehouse in it's raw form..
...Why is this important? In an ETL process if something goes wrong it's very hard to debug if the issue happened in the "T" or the "L". It was also harder to build these pipelines. With ELT, tools like @fivetran allow you to point & click to connect source data to your warehouse
The big trend here? The data warehouse is starting to subsume the data lake, and the default is becoming: "just send all data to Redshift / Snowflake." Again, the barriers of storing / collecting data is going way down
2. The importance of the cloud data warehouse. We already talked about this, but the one incremental point I want to make is the power of the compute within the warehouse went way up, and the cost of that compute went way down...
...this is fundamentally what enabled the "T" in ELT to happen within the warehouse. The compute horsepower of a Snowflake / Redshift made it possible
3. So who's driving these transformations? Tools like @getdbt @fishtowndata. The big technology advancement of the open source project dbt was representing these data transformations as code (SQL). It allowed anyone who knows SQL (business analysts) to author the transformations
Prior, the transformations were done with custom Python code by data engineers, or GUI based ETL tools. These took forever to build, were inflexible / hard to scale, and a black box
So in summary, the major platforms enabling the rise of DataOps are:

1. Data Movement: @fivetran
2. Data storage / compute: @SnowflakeDB @awscloud @GCPcloud
3. Data Transformations: @getdbt
If you think about what these technologies allow: to get data into a warehouse you just point and click data sources to their destination. The connectors are pre-built. You don't have to manage an on-prem database. And the data transformations are represented as basic SQL
And here is my KEY point (I get I've buried the lead a bit, but I believe the setup is important): Data Engineers used to manage all of the complexities of moving, storing, and transforming data. A lot of it was built with custom code / Python, and managed MANUALLY 🤯
Today, with the powerful tools I listed above, the data ecosystem can be managed with turnkey tools, removing the need for a lot of the complex work the data engineers handled previously....
...Instead of the BI analysts requesting the data engineer build them a pipeline of data, they can point & click with Fivetran and get data loaded into Snowflake, and write some SQL with dbt to get a materialized view (subset of data) to efficiently run their query against
After quite the lead in I hope that it's clear now how the business analysts (like BI analysts) can now carry out the functionality of data engineers to access data in a turnkey, self-serve manner. I'm describing these analysts as the "Analytica Engineer" and the process DataOps
So why is this important, and why am I excited about DataOps this year?

When data access becomes democratized and self-serve in nature, the need for new tools to manage this "modern data stack" goes up. I think we'll see a TON of huge companies built in the following categories:
1. I think we'll see the data catalogue / lineage systems re-invented. To access data you need to know where it's located / where it came from

2. Monitoring the quality of the data (and pipelines) will become increasingly important in self-serve settings. "Datadog for data"
3. A wave of modern BI tools

4. Governance becomes a bigger deal. With self-serve access, how can you govern who should have access to what?

5. Metadata gains increased importance

6. The way data pipelines were orchestrated (ie Airflow) will completely evolve
6. The "endpoints" for data coming out warehouses will grow. Right now the two primary places data goes from the warehouse are BI ML tools. I think we'll see an explosion of data going into new places, like back into SaaS apps

7. And many more we can't even imagine now

More from Tech

Aditya Todmal
Aditya Todmal
@AdityaTodmal
The absolute best 15 scanners which experts are using.

Got these scanners from the following accounts:

1. @Pathik_Trader
2. @sanjufunda
3. @sanstocktrader
4. @SouravSenguptaI
5. @Rishikesh_ADX

Share for the benefit of everyone.

Listing all scanners from @Pathik_Trader Sir first.

1. Open

2. 80-20 Reversal Setup


3. Fake

4. Trapping Inside Bar (Positional Setup)
TECH , STOCKLEARNINGS , STOCKS
foone
foone
@Foone
you know how Atari called the Jaguar as a "64bit system", because it had two 32bit CPUs? (or because it had a 64bit bus, but let's not worry about that)
we should apply that logic to the current generation of consoles.
So the Switch is a 256bit console. It's the Nintendo 256.

The Xbox Series X? terrible name.
Better call it the "Xbox 512 Blu-ray".
like the Jaguar CD, the Sega CD... the good old days when consoles had their media type in the name.

Boringly the PS5 is also a Playstation 512 Blu-ray, because both of them are using the same AMD Zen 2 CPU.

The Switch one is weeeird. It's built on an Nvidia Tegra X1, which is an 8 core 64-bit CPU, so that should be 512, right?

Well... no. Four of them can't be used. And later documentation doesn't even mention them.

it's some weird system where the system-on-a-chip has four Cortex-A57 cores and four Cortex-A53 cores but only the A57s are usable?
TECH
Advertisement
Kelly Ellis
Kelly Ellis
@justkelly_ok
So, I want to talk about how sexual harassment and fair pay are linked at Google (and beyond), because I think that's an angle that isn't being highlighted enough in the coverage of these walk-outs. I'm going to frame it largely around my personal experience there. Thread.

When I was sexually harassed by the director of the area I was working in, I was afraid to report it because I was worried that "getting him in trouble" would result in the subtle retaliation of missed leadership opportunities.

I wanted to continue working on the team I was on, because I'd gained a lot of very deep knowledge and expertise in that area, as well as reputation and camaraderie with the other folks working in that area. I didn't want to make the situation more "difficult."

To get promoted at Google, several need to happen: 1. you need opportunities for ownership and leadership above your current level (basically, opportunities to show you're working at the next level you're trying to get promoted to). The work you're "assigned" has a big impact.

2. You need glowing reviews from peers, *at or particularly above the level you're hoping to get promoted to.* Basically, you need people a lot more senior than you to say you're doing awesome work.
TECH , SOCIETY
Paul Matzko
Paul Matzko
@PMatzko
The New York Times article about SlateStarCodex is finally out and it is...bad. There's a lot I could parse, but let me just walk you through one paragraph that is so misleading as to be

Take the first sentence of this paragraph. Now, technically the clause--"who proposed a link between race and IQ"--could simply modify "Murray" and have nothing to do w/ SSC.


But 99% of readers are going to assume that the clause actually defines SSC's alignment with Murray. In other words, the author is strongly implying that SSC shares Murray's racist beliefs.

And that would be big(!)...if it were at all true. It is not. Indeed, if you go to the hyperlink, you'll find that SSC's purported "alignment" has nothing to do with Murray's "Bell Curve."

What SSC & Murray agree about is that poverty is partly hereditable and thus very sticky, so much so that the proposition job retraining programs will meaningfully address mass economic disruption is a pie-in-the-sky fantasy.
TECH
Devdutta
Devdutta
@Devdutta96
Please please please take your cyber security seriously even if you don’t consider yourself a high risk person. Here are a few easy things to start with. 1/n

Switch from WhatsApp to Signal and enable disappearing messages. Most of us cannot stop using WhatsApp entirely but at least turn off back ups to Google Drive or iCloud. 2/n

Do not have private conversations using the messaging service on any social media platform like Facebook, Instagram or Twitter. 3/n

Switch from Gmail to ProtonMail or any other secure email service provider. 4/n


Switch from Google Docs to CryptPad or RiseUp Pad.
CYBER SECURITY , IT PRIVACY , CYBER SECURITY STEPS

You May Also Like

Chris Herd
Chris Herd
@chris_herd
I've spoken to 1,500+ people about remote work in the last 9 months

A few predictions of what is likely to emerge before 2030

[ a thread ] 💻🏠🌍

🚜 Rural Living: World-class people will move to smaller cities, have a lower cost of living & higher quality of life

These regions must innovate quickly to attract that wealth. Better schools, faster internet connections are a must

⏰Asynchronous Work: Offices are instantaneous gratification distraction factories where synchronous work makes it impossible to get stuff done

Tools that enable asynchronous work are the most important thing globally remote teams need. A lot of startups will try to tackle this

⚽️Hobbie Renaissance: Remote working will lead to a rise in people participating in hobbies and activities which link them to people in their local community

This will lead to deeper, more meaningful relationships which overcome societal issues of loneliness and issolation

🌍Diversity & Inclusion: The most diverse and inclusive teams in history will emerge rapidly

Companies who embrace it have a first-mover advantage to attract great talent globally. Companies who don't will lose their best people to their biggest competitors
REMOTE WORK
\u092c\u094d\u0930\u093e\u0939\u094d\u092e\u0923\u094b\u0902 \u0915\u093e \u0907\u0924\u093f\u0939\u093e\u0938\U0001f441\ufe0f\U0001f441\ufe0f
ब्राह्मणों का इतिहास👁️...
@BRAHMIN_0052
MOHIYAL_BRAHMIN_KING
RAJA_PORUS
King_Porus_History

1)King Porus was a famous Indian king, his real name was #Purushotama. He belonged to PURU tribe mentioned in #RIG_VEDA, the Ancient books of India.


2) Porus ruled the Punjab region and expanded his kingdom between the Jhelum River and Chenab River in the Indian Subcontinent. He was a legendary warrior and more war skills. He made the army most powerful with war skills. Even Alexander had also made difficulty to defeat Porus.

3)Word of Mouth Oral tradition in #Mohyal_Brahmin families since ancient times has kept the Great Saga of their Great Elder known to them as Maharaja “ #Puru” alive.

4) When #Alexander the Great invaded India in 326 BC he was challenged near Taxila by a mere chieftain of the Area. He was King #Porus, a #Vaid_Mohyal. After his encounter with Porus, Alexander met with more resistance near Multan where the forces were predominantly #Mohyal.

5)The Bali rulers of Multan fiercely engaged with Alexander's army in skirmishes in which many senior Generals died. Later during the long and bitter fighting in #Multan, Alexander got hurt by an arrow which pierced his chest ,
RELIGION
Advertisement
Khatvaanga
Khatvaanga
@khatvaanga
Starting my 2nd thread on Sringeri Parampara. This will cover the Jagadgurus after Sri Vidyaranya to Sri Shivabhinava Nrisimha Bharati.

It is really very tough to match the brilliance and achievements of Vidyaranya. After whom came 15 - Chandrashekara Bharati [1386-1389].

The next Acharya was Nrisimha Bharati [1389-1408]. Harihara raya found a town for this Acharya - Narasimhapuram. Acharya helped the king by guiding him in political affairs. He also initiated the kind with mahamantras for spiritual development.

Subsequent Acharyas were
15-Puroshottama Bharati [1408 - 1448]
16-Shankara Bharati[1448 - 1455]
17-Chandrasekhara Bharati II [1455 - 1464]
18- Nrisimha Bharati II [1464 - 1479]
19-Puroshottama Bharati II [1479 - 1517]

20-Ramachandra Bharati [1517 - 1560]. Sadashiva Raya was emporer of Vijayanagara during this time.
21-Nrisimha Bharati III [1560 - 1573]
22-Nrisimha Bharati IV [1573 - 1576]
23-Nrisimha Bharati V [1576 - 1600]

24-Abhinava Nrisimha Bharati [1600-1623]. When he visited Malahanikareswara temple for first time he noticed there was no Vinayaka there so he took piece of turmeric and drew outline of Ganesha on a pillar and offered puja.
ALL
Professor
Professor
@DillikiBiili
So friends here is the thread on the recommended pathway for new entrants in the stock market.

Here I will share what I believe are essentials for anybody who is interested in stock markets and the resources to learn them, its from my experience and by no means exhaustive..

First the very basic : The Dow theory, Everybody must have basic understanding of it and must learn to observe High Highs, Higher Lows, Lower Highs and Lowers lows on charts and their

Even those who are more inclined towards fundamental side can also benefit from Dow theory, as it can hint start & end of Bull/Bear runs thereby indication entry and exits.


Next basic is Wyckoff's Theory. It tells how accumulation and distribution happens with regularity and how the market actually

Dow theory is old but
SCREENERS , ALL , MARKETESSENTIALS
Vibhu Vashisth
Vibhu Vashisth
@VIBHU_Tweet
Did you know?

When Saudi Arabia's King visited India in 1955, PM Nehru ordered all the Hindu Temples and Idols of Hindu Deities that came in Saudi King's way to be covered up.
Our temples are the architectural marvels in their own league.


Each of our temples tells the story of our rich history and cultural heritage. Was Nehru that ashamed of his own nation's heritage that he had to cover them up or there was some other reason? The answer would be best known to him only.

PM Nehru invited the then Saudi King Saud bin Abdul Aziz to India who accepted the invitation and planned a 17 day long official visit to India in 1955. Saudi King had brought a large delegation with him and he had to visit various Indian cities like Mumbai, Hyderabad, Agra,


Aligarh, Shimla, Mysore and Varanasi. However, when Saudi King reached Varanasi, Nehru ordered to cover all the temples and Idols with curtains that came in the way of the Royal Convoy. During his stay in Varanasi, Govt buildings were painted with the words "Kalma Tayyiba".


'Kalma Tayyiba' means 'There is not God but only Allah and Mohammed is his messenger'. In addition to it, Pooja and Aartis were also stopped at temples. I can call it the appeasement politics of the highest order and Nehru was indeed the greatest appeaser of all times.
ALL