BC TECH
Saved by @SteveeRogerr

A trend I'm excited for this year: DataOps & the Analytical Engineer

~10 years ago DevOps was born. The role of system admins and developers merged. Infrastructure became self-serve

Today the role of data engineers and business analysts are merging. Data is becoming self-serve

 Data infrastructure is becoming so powerful that the tools today allow non-technical folks to carry out the once complicated / custom code/ huge backlog jobs of data engineers.

Before getting into what this means, let's first discuss how we got here
Before 2012 the data world was dominated by transactional (OLTP) databases like PostgreSQL, MySQL, etc and analytical (OLAP) databases like Oracle, Netezza

Tools like Informatica / Talend were used to batch load (ETL) data into these databases, Tableau used to visualize
As you can imagine, there was heavy engineering work to manage the environment...

Then in 2013 AWS released their cloud data warehouse Redshift, and it was a game changer. Snowflake was founded in 2012, but didn't really pick up steam until a few years later (around 2016)
So why was Redshift a big deal?

1. It was the first cloud-native OLAP warehouse. It reduced the TCO of an OLAP database by orders of magnitude.

2. Speed of processing analytical queries increased dramatically
3. And later on (Snowflake pioneered this) they separated compute & storage. In overly simplified terms, this meant customers could scale their storage and compute resources independently of one another. This was a huge deal

What does this all mean? An EXPLOSION of data
The barriers to maintain a database were completely broken down, and the amount of data that was sent to Redshift / Snowflake / BigQuery skyrocketed.

Now, after this point we still weren't ready for DataOps / Analytical Engineers. What did it take to get there?
In my opinion there were 3 major technologies / shifts that happened that have given rise to DataOps:
1. The shift from ETL to ELT (extract-transform-load to extract-load-transform). Data used to be transformed (joined, aggregated, cleaned, etc) in motion while being loaded into the warehouse. Now, data is being loaded into the warehouse in it's raw form..
...Why is this important? In an ETL process if something goes wrong it's very hard to debug if the issue happened in the "T" or the "L". It was also harder to build these pipelines. With ELT, tools like @fivetran allow you to point & click to connect source data to your warehouse
The big trend here? The data warehouse is starting to subsume the data lake, and the default is becoming: "just send all data to Redshift / Snowflake." Again, the barriers of storing / collecting data is going way down
2. The importance of the cloud data warehouse. We already talked about this, but the one incremental point I want to make is the power of the compute within the warehouse went way up, and the cost of that compute went way down...
...this is fundamentally what enabled the "T" in ELT to happen within the warehouse. The compute horsepower of a Snowflake / Redshift made it possible
3. So who's driving these transformations? Tools like @getdbt @fishtowndata. The big technology advancement of the open source project dbt was representing these data transformations as code (SQL). It allowed anyone who knows SQL (business analysts) to author the transformations
Prior, the transformations were done with custom Python code by data engineers, or GUI based ETL tools. These took forever to build, were inflexible / hard to scale, and a black box
So in summary, the major platforms enabling the rise of DataOps are:

1. Data Movement: @fivetran
2. Data storage / compute: @SnowflakeDB @awscloud @GCPcloud
3. Data Transformations: @getdbt
If you think about what these technologies allow: to get data into a warehouse you just point and click data sources to their destination. The connectors are pre-built. You don't have to manage an on-prem database. And the data transformations are represented as basic SQL
And here is my KEY point (I get I've buried the lead a bit, but I believe the setup is important): Data Engineers used to manage all of the complexities of moving, storing, and transforming data. A lot of it was built with custom code / Python, and managed MANUALLY 🤯
Today, with the powerful tools I listed above, the data ecosystem can be managed with turnkey tools, removing the need for a lot of the complex work the data engineers handled previously....
...Instead of the BI analysts requesting the data engineer build them a pipeline of data, they can point & click with Fivetran and get data loaded into Snowflake, and write some SQL with dbt to get a materialized view (subset of data) to efficiently run their query against
After quite the lead in I hope that it's clear now how the business analysts (like BI analysts) can now carry out the functionality of data engineers to access data in a turnkey, self-serve manner. I'm describing these analysts as the "Analytica Engineer" and the process DataOps
So why is this important, and why am I excited about DataOps this year?

When data access becomes democratized and self-serve in nature, the need for new tools to manage this "modern data stack" goes up. I think we'll see a TON of huge companies built in the following categories:
1. I think we'll see the data catalogue / lineage systems re-invented. To access data you need to know where it's located / where it came from

2. Monitoring the quality of the data (and pipelines) will become increasingly important in self-serve settings. "Datadog for data"
3. A wave of modern BI tools

4. Governance becomes a bigger deal. With self-serve access, how can you govern who should have access to what?

5. Metadata gains increased importance

6. The way data pipelines were orchestrated (ie Airflow) will completely evolve
6. The "endpoints" for data coming out warehouses will grow. Right now the two primary places data goes from the warehouse are BI ML tools. I think we'll see an explosion of data going into new places, like back into SaaS apps

7. And many more we can't even imagine now

More from Tech

Matthew Panzarino
Matthew Panzarino
@panzer
New iPad Pro has Face ID, thinner bezels, squared off edges. A new touch sensitive Pencil that magnetically attaches to the edges. Thin enough to have a camera bump. Uses the XR's LCD cornering techniques.


The new 12.9" iPad Pro is physically smaller than the old one with the same screen size. It's roughly the same size as an 8.5 x 11 piece of paper. 5.9mm thick.

10 billion transistors in the A12X. 8-core CPU. 7-Core GPU. 35% faster single-core performance. 1000x faster GPU. iPad Pro is Xbox One S class GPU performance in "94% smaller" package, Ternus jokes.

Apple's silicon team is just dropping bombs every event.

live shot of Apple's silicon team


iPad Pro gets USB-C for up to 5K external displays and enables charging out to iPhones. Wild.
TECH
Venkatesh Rao
Venkatesh Rao
@vgr
There are only 3 real challenges to free speech at the tech level a) the need for CDNs at scale b) distortions of net neutrality c) nation-state firewalls.

All other claims are about social governance on private sites, where applying free speech doctrine is a stretch at best

Baseline free speech has expanded modestly since 1776 or whatever your reference point is.

About 20% beyond core case of government-critical speech in formally public spaces and media I'd say.

You can (for example) have more kinds of conversations in a bar without getting punched or kicked out by other patrons or management. You can teach more things in colleges than 100 years ago.

That's a genuine de facto increase in practical, informal free speech/expression.

But online free "speech" is, as @reneed has pointed out, not an expansion of free speech at all, but an expansion of reach. It is like the secondary freedom of the press to print as many copies as they can afford, not primary freedom to say what they want

If a newspaper wants to print more copies it has buy more printing equipment and front more upfront costs. Reach for a paper publication is not "free", but a function of desired circulation, which in turn is a function of demand prediction etc. But that logic is obscured online.
TECH
Advertisement
foone
foone
@Foone
you know how Atari called the Jaguar as a "64bit system", because it had two 32bit CPUs? (or because it had a 64bit bus, but let's not worry about that)
we should apply that logic to the current generation of consoles.
So the Switch is a 256bit console. It's the Nintendo 256.

The Xbox Series X? terrible name.
Better call it the "Xbox 512 Blu-ray".
like the Jaguar CD, the Sega CD... the good old days when consoles had their media type in the name.

Boringly the PS5 is also a Playstation 512 Blu-ray, because both of them are using the same AMD Zen 2 CPU.

The Switch one is weeeird. It's built on an Nvidia Tegra X1, which is an 8 core 64-bit CPU, so that should be 512, right?

Well... no. Four of them can't be used. And later documentation doesn't even mention them.

it's some weird system where the system-on-a-chip has four Cortex-A57 cores and four Cortex-A53 cores but only the A57s are usable?
TECH
Santiago
Santiago
@svpino
Here is a simple example of a machine learning model.

I put it together a long time ago, and it was very helpful! I sliced it apart a thousand times until things started to make sense.

It's TensorFlow and Keras.

If you are starting out, this may be a good puzzle to solve.


The goal of this model is to learn to multiply one-digit
TECH
Trond Bjor\xf8y
Trond Bjorøy
@trondbjoroy
1/12 @HathorNetwork offers a unique set of features for both builders and users. Some well known ones, others not so much. Let’s look at why you might want to consider Hathor as the layer 1 solution for your project. $HTR @HTRFDT #BuildOnHathor

2/12 Easy tokenization is the obvious one. You create your own token in less than a minute, no coding required.

3/12 As opposed to ERC tokens, your custom Hathor token is a layer 1 token. It inherits the same features and security as the native token, $HTR, effectively giving your token the same level of security as the #Bitcoin $BTC blockchain.

4/12 Hathor Network also passed their security audit by @SlowMist_Team so you can feel safe knowing that your L1 token fulfills the same security requirements as $HTR.

5/12 By configuring built-in nano contracts you will be able to add logic to your token. No coding required. Native nano contracts also mitigate the risk of bugs and security issues that comes with custom smart contracts development on other platforms. $HTR
TECH

You May Also Like

Enigma
Enigma
@ainittomai
The Eye of Horus. 1/*


I believe that @ripple_crippler and @looP_rM311_7211 are the same person. I know, nobody believes that. 2/*

Today I want to prove that Mr Pool smile faces mean XRP and price increase. In Ripple_Crippler, previous to Mr Pool existence, smile faces were frequent. They were very similar to the ones Mr Pool posts. The eyes also were usually a couple of "x", in fact, XRP logo. 3/*


The smile XRP-eyed face also appears related to the Moon. XRP going to the Moon. 4/*


And smile XRP-eyed faces also appear related to Egypt. In particular, to the Eye of Horus. https://t.co/i4rRzuQ0gZ 5/*
CULTURE
Jaya_Upadhyaya
Jaya_Upadhyaya
@Jayalko1
JWALAMUKHI TRIPURASUNDARI, UTHANAHALLI, MYSURU (KAR)
#bharatmandir

Jwalamukhi Tripursundari is believed to be the sister of Devi Chamundeshwari, the presiding deity of Mysuru
It is also that she represents the flaming mouth of Jalandhara, the demon whom Shiva crushed to death.


When Devi Chamundeswari was fighting Mahishasura, she observed that from every drop of blood that spilled out of Mahishasura, another demon was born. So she created Tripurasundari to ensure that no blood dripped from Mahishasura.


Tripura Sundari means beauty of the three worlds.
The murti was installed during Wodeyar rule. An utsava murti made of pancha loha is kept in the sanctum sanctorum.
Another shrine here is dedicated to Siddeshwara / Shiva surrounded by a beautiful wooden Mantapa.
@GunduHuDuGa
ALL
Advertisement
Brian Cates
Brian Cates
@drawandstrike
A lot of people don't understand that Trump & his incoming team didn't face a normal transition period, like it was over a in a matter of a few weeks or months.

The DOJ transition period JUST ENDED when Sessions left. Almost 2 exact years to get it there.

Pompeo at the CIA took over a year before he was done and ready to shift over to the State Department.

All kinds of stuff has been going on behind the scenes nobody leaks or talks about.

We only had the barest mention of the fact the Clinton Email investigation restarted

Nobody leaked in a year that Huber was deep diving into the Clinton Foundation.
Keep that in mind as all the usual suspects continue to INSIST nothing is happening/going to happen.
POLITICS
Simon DeDeo
Simon DeDeo
@SimonDeDeo
One thing that goes unremarked about Wikipedia: the vast majority of the content was written over 12 years ago—which means (given turnover) that the editors dominant today were not directly involved in that creation.

Put another way, the editors who built the dominant nodes in this network...


...have little overlap with the ones who made this much more recent managerial flowchart.


Internet time runs at hundred-fold speed—the difference between the people who painted what's in the Uffizi, and the people in charge of keeping those paintings from deteriorating. Very different tasks, and (one presumes) very personalities as well. @PaulSkallas?
ALL
\U0001f1ee\U0001f1f3Sonal \U0001f1ee\U0001f1f3
🇮🇳Sonal 🇮🇳...
@Dharma_Vaarta
Bharat: Advancement of earth’s earliest civilization

Glorious Vedic Architecture and Temple

Architecture in India was an extremely rich form of expression and highly developed science, with a deep heritage of spirituality in it. #FreeHinduTemple


The centre of all architectural development was the temple and temple or Mandir was the earthly home of God, and like the launching pad for the devotee’s consciousness to soar toward the Divine and the spiritual world.


The murti or deity of the Divine would be placed in the temple sanctum, around which all temple activity would revolve.

Such buildings were often elaborately decorated with stone, wood, plaster, etc. and carved or cut to depict the stories from the Puranas or other sacred texts


Temples were always the centre of religious, social and educational activities, practically more so in
ancient times than today.

During a span of more than two thousand years, various dynasties were known for constructing glorious temples,


some of which include the Mauryan, Ashoka, Gupta, Satavahana, Chedi, Chandella, Chalukya, Solanki, Pallava, Shola, Hoysala, Vijayanagar, and others.

Many of these temples remain beautiful architectural wonders today.
ALL , HISTORY