Before getting into what this means, let's first discuss how we got here
A trend I'm excited for this year: DataOps & the Analytical Engineer
~10 years ago DevOps was born. The role of system admins and developers merged. Infrastructure became self-serve
Today the role of data engineers and business analysts are merging. Data is becoming self-serve
Before getting into what this means, let's first discuss how we got here
Tools like Informatica / Talend were used to batch load (ETL) data into these databases, Tableau used to visualize
Then in 2013 AWS released their cloud data warehouse Redshift, and it was a game changer. Snowflake was founded in 2012, but didn't really pick up steam until a few years later (around 2016)
1. It was the first cloud-native OLAP warehouse. It reduced the TCO of an OLAP database by orders of magnitude.
2. Speed of processing analytical queries increased dramatically
What does this all mean? An EXPLOSION of data
Now, after this point we still weren't ready for DataOps / Analytical Engineers. What did it take to get there?
1. Data Movement: @fivetran
2. Data storage / compute: @SnowflakeDB @awscloud @GCPcloud
3. Data Transformations: @getdbt
When data access becomes democratized and self-serve in nature, the need for new tools to manage this "modern data stack" goes up. I think we'll see a TON of huge companies built in the following categories:
2. Monitoring the quality of the data (and pipelines) will become increasingly important in self-serve settings. "Datadog for data"
More from Tech
BREAKING: @CommonsCMS @DamianCollins just released previously sealed #Six4Three @Facebook documents:
Some random interesting tidbits:
1) Zuck approves shutting down platform API access for Twitter's when Vine is released #competition
2) Facebook engineered ways to access user's call history w/o alerting users:
Team considered access to call history considered 'high PR risk' but 'growth team will charge ahead'. @Facebook created upgrade path to access data w/o subjecting users to Android permissions dialogue.
3) The above also confirms @kashhill and other's suspicion that call history was used to improve PYMK (People You May Know) suggestions and newsfeed rankings.
4) Docs also shed more light into @dseetharaman's story on @Facebook monitoring users' @Onavo VPN activity to determine what competitors to mimic or acquire in 2013.
https://t.co/PwiRIL3v9x
Some random interesting tidbits:
1) Zuck approves shutting down platform API access for Twitter's when Vine is released #competition
2) Facebook engineered ways to access user's call history w/o alerting users:
Team considered access to call history considered 'high PR risk' but 'growth team will charge ahead'. @Facebook created upgrade path to access data w/o subjecting users to Android permissions dialogue.
3) The above also confirms @kashhill and other's suspicion that call history was used to improve PYMK (People You May Know) suggestions and newsfeed rankings.
4) Docs also shed more light into @dseetharaman's story on @Facebook monitoring users' @Onavo VPN activity to determine what competitors to mimic or acquire in 2013.
https://t.co/PwiRIL3v9x
On press call, Zuckerberg says FB users "naturally engage more with sensational content" that comes close to violating its rules. Compares it to cable TV and tabloids, and says, "This seems to be true regardless of where we set our policy lines."
Zuckerberg says FB is in the process of setting up a "new independent body" that users will be able to appeal content takedowns to. Sort of like the "Facebook Supreme Court" idea he previewed earlier this year.
Zuckerberg: "One of my biggest lessons from this year is that when you connect more than 2 billion people, you’re going to see the good and bad of humanity."
This is how Facebook says it's trying to change the engagement pattern on its services. https://t.co/3p0PGc912o
.@RebeccaJarvis asks Zuckerberg if anyone is going to lose their job over the revelations in the NYT story. He dodges, says that personnel issues aren't a public matter, and that employee performance is evaluated all the time.
Zuckerberg says FB is in the process of setting up a "new independent body" that users will be able to appeal content takedowns to. Sort of like the "Facebook Supreme Court" idea he previewed earlier this year.
Zuckerberg: "One of my biggest lessons from this year is that when you connect more than 2 billion people, you’re going to see the good and bad of humanity."
This is how Facebook says it's trying to change the engagement pattern on its services. https://t.co/3p0PGc912o
.@RebeccaJarvis asks Zuckerberg if anyone is going to lose their job over the revelations in the NYT story. He dodges, says that personnel issues aren't a public matter, and that employee performance is evaluated all the time.
You May Also Like
I’m torn on how to approach the idea of luck. I’m the first to admit that I am one of the luckiest people on the planet. To be born into a prosperous American family in 1960 with smart parents is to start life on third base. The odds against my very existence are astronomical.
I’ve always felt that the luckiest people I know had a talent for recognizing circumstances, not of their own making, that were conducive to a favorable outcome and their ability to quickly take advantage of them.
In other words, dumb luck was just that, it required no awareness on the person’s part, whereas “smart” luck involved awareness followed by action before the circumstances changed.
So, was I “lucky” to be born when I was—nothing I had any control over—and that I came of age just as huge databases and computers were advancing to the point where I could use those tools to write “What Works on Wall Street?” Absolutely.
Was I lucky to start my stock market investments near the peak of interest rates which allowed me to spend the majority of my adult life in a falling rate environment? Yup.
Ironies of Luck https://t.co/5BPWGbAxFi
— Morgan Housel (@morganhousel) March 14, 2018
"Luck is the flip side of risk. They are mirrored cousins, driven by the same thing: You are one person in a 7 billion player game, and the accidental impact of other people\u2019s actions can be more consequential than your own."
I’ve always felt that the luckiest people I know had a talent for recognizing circumstances, not of their own making, that were conducive to a favorable outcome and their ability to quickly take advantage of them.
In other words, dumb luck was just that, it required no awareness on the person’s part, whereas “smart” luck involved awareness followed by action before the circumstances changed.
So, was I “lucky” to be born when I was—nothing I had any control over—and that I came of age just as huge databases and computers were advancing to the point where I could use those tools to write “What Works on Wall Street?” Absolutely.
Was I lucky to start my stock market investments near the peak of interest rates which allowed me to spend the majority of my adult life in a falling rate environment? Yup.
A list of cool websites you might now know about
A thread 🧵
1) Learn Anything - Search tools for knowledge discovery that helps you understand any topic through the most efficient
2) Grad Speeches - Discover the best commencement speeches.
This website is made by me
3) What does the Internet Think - Find out what the internet thinks about anything
4) https://t.co/vuhT6jVItx - Send notes that will self-destruct after being read.
A thread 🧵
1) Learn Anything - Search tools for knowledge discovery that helps you understand any topic through the most efficient
2) Grad Speeches - Discover the best commencement speeches.
This website is made by me
3) What does the Internet Think - Find out what the internet thinks about anything
4) https://t.co/vuhT6jVItx - Send notes that will self-destruct after being read.
