BC SOCIETY
Saved by @jacobhtml

0/ When large eng orgs rely on metrics for both monitoring *and* observability, they struggle with cardinality.

This is a thread about “the two drivers of cardinality.” And which one of those we should kill. :)

🧵👇

 1/ Okay, first off: “what is cardinality, anyway?” And why is it such a big deal for metrics?

“Cardinality” is a mathematical term: it’s *the number of elements in a set*... boring! So why tf does anybody care??

Well, because people think they need it, then suddenly, "$$$$$$$."
2/ When a developer inserts a (custom) metric to their code, they might think they’re just adding, well, *one metric*. …
3/ … But when they add “tags” to that metric – like , , or (shiver) – they are actually creating a *set* of metric time series, with the *cardinality* of that set being the total number of unique combinations of those tags.
4/ The problem is that some of those tags have many distinct values.

E.g., when I was working on Monarch at Google, there was a single gmail metric that became over 300,000,000 (!!!) distinct time series. In a TSDB, that cardinality is the unit of cost.

So again, “$$$$$$$.”
5/ Okay, so that’s why people care about metrics cardinality. Now, what are the two *drivers* of that cardinality?

A) Monitoring: more detailed *health measurements*
B) Observability: more detailed *explanations of changes*

Let’s take these one at a time…
6/ First, “More Detailed Health Measurements” (monitoring):

Consider an RPC metric that counts errors. You need to independently monitor the error rate for different methods. And so – voila – someone adds a “method” tag, and now there’s 10x the cardinality for that metric.
7/ … And also 10x the cost. But that’s justifiable, as there’s a business case for independently monitoring the reliability of distinct RPC methods.

Put another way, you might rightly have different error budgets for different RPC methods, so their statistics must be separable.
8/ Bottom line: When it comes to “measuring health,” we often *need* cardinality in order to hone in on the signals we actually care about. Increasing cardinality to *proactively* monitor the signals we care most about is usually a worthwhile tradeoff.
9/ Now what about using cardinality for “More Detailed Explanations of Changes?”

This is the real culprit! And, frankly, should be abolished. :) Metrics cardinality is the wrong way to do observability – to explain changes.

More on that…
10/ Say your monitoring tells you that there’s a problem with a critical symptom – e.g., you’re suddenly burning through an SLO error budget at an unsustainable clip.
11/ After a painful outage, say you realize a single customer DOSed your service. So someone adds a `customer` tag “for next time.”

But this is unsustainable: each incident reveals a new failure mode, devs keep adding tags, and before long your metrics bill is out of control.
12/ The problem, in a nutshell:

Distributed systems can fail for a staggeringly large number of reasons. *You can't use metrics cardinality to isolate each one.*
13/ How *should* we explain changes to production systems?

Understanding change is the central problem of observability. Effective workflows might *start* with metrics, but they must pivot towards a multi-telemetry, multi-service guided analysis.
14/ So, to sum up: spend your limited cardinality budget on *monitoring*, and then look for observability that (a) naturally explains changes and (b) relies on transactional data sources that do not penalize you for high/unbounded cardinality.
15/ PS: For more on how to distinguish monitoring and observability, see this thread: https://t.co/2RHs36Nknc

PPS: If you’d like to discuss/debate/request-more-detail any of the above, reply to this thread or DM me!

More from Society

Noah Smith
Noah Smith
@Noahpinion
Remember: Just because an idea is old doesn't mean it's good.

Knowing whether old ideas are more likely to be good requires understanding whether conditions have changed in important ways.

Horse archery was an amazing method of warfare for over a thousand years...then people invented guns, and suddenly this great idea that had stood the test of time became obsolete.

Then again, when underlying conditions don't change much, tried-and-true approaches are probably better.

"Learning the lessons of history" usually just means assuming ergodicity and stationarity in an informal time-series model...

(end)
SOCIETY
Miss C
Miss C
@Credible_Intel
1. Most people are aware of the fail doxes posted by Jamilla Moor on @Snakeeater36 and myself. I'm going to go into the details (in layman terms) of exactly how a specific group of duhnonymous idiots came up with these fail doxes & I'll prove that they're wrong.

2. @Snakeeater36 posted this screenshot on his TL. He had exposed the page source code on tweet which included an image. Anyone can do this in Chrome by opening a tweet, clicking on the image, then right click on any open space on the screen, select "inspect", then "network".

3. In fact I did the same by pulling up a different tweet and image. You'll see the IP highlighted in yellow and the region Dallas TX. It's not my IP. It belongs to Akamai which is identified. Why is that important.......


4. Well Akamai is a Content Delivery Network also known as a CDN. A CDN, is a geographically distributed network of proxy servers (make note of the term "proxy" servers) and their data centers. More later on why this is important relative to the fail doxes.

5. CDNs allow companies like Twitter to provide better performance in uploading content, like images, by distributing the service spatially relative to end users. Rather than pushing content directly (left), Twitter uses CDN networks (right) to push/pull images from end users.
SOCIETY
Advertisement
Dr Fish Philosopher Todd (Dr FPT)\U0001f41f
Dr Fish Philosopher To...
@ZoeSTodd
1. I was walking in the woods yesterday and I had an epiphany. It wasn’t one I wanted to have. I was thinking about today’s class, where we are reading Kimmerer’s ‘Gathering Moss’. I’ve taught Braiding Sweetgrass in multiple classes every year since 2016. It’s eminently teachable

2. Both books are beautifully written. Students uniformly enjoy them as assigned reading. They do a great job communicating to non-Indigenous folks the contours of basic Indigenous relationships to more-than-human beings. They’re wildly popular with white readers.

3. I’ve found Braiding Sweetgrass to be a great entry point for non-Indigenous folks to Indigenous scholarship. But this is where yesterday’s epiphany comes in. Neither Braiding Sweetgrass or Gathering Moss tends to the deep deep lineages of Indigenous scholarship in Canada or US

4. As the popularity of ‘Braiding Sweetgrass’ has soared, I’ve seen more and more white folks wax on about ‘braiding knowledge’. But they often don’t know any of the lineages of these concepts beyond what is presented in this one book. They confidently run with this.

5. Kimmerer does a commendable job of weaving in her own experiences, her family stories+nods to some very important knowledge keepers who deserve to be celebrated for their work. But Braiding Sweetgrass does not tend to the citational politics Sara Ahmed & others teach us about
SOCIETY
Jerusalem
Jerusalem
@JerusalemDemsas
For Black History Month, Vox is publishing a big series on rethinking policy for Black America.

I wrote 3,200 words on eradicating exclusionary zoning.

It's a 3 step formula: Persuade, incentivize, and if all else fails, SUE THE SUBURBS.

Thread

We know that exclusionary zoning is locking millions out of opportunity.

Work by @OppInsights, Richard Rothstein, Ta-Nehisi Coates, @nhannahjones, @RickKahlenberg, Chang-Tai Hsieh, Enrico Moretti and @ProfSchleich details the economic impact of these discriminatory policies.


51 years after the Fair Housing Act, it's never been seriously enforced. It's time to change that.

First things first, Biden has work to do to convince whoever can be convinced that there is a racial, environmental, and economic imperative to undo residential segregation.


Use whatever explanation works!

For sympathetic progressives Biden should link racial and environmental justice to ending exclusionary zoning.

For everyone else, he can simply make an economic

Second, there are many tools to incentivize localities that rely on federal dollars to reform their zoning codes.

There is a ton of federal money that can be conditioned on reforming exclusionary zoning laws. @ebwhamilton has some good ideas on this front.
SOCIETY
Derik Chica
Derik Chica
@DerikChica
Yesterday I refused unsafe work. This is how it went.

#onted #onpoli #osstf

1st, to the haters. Since when is it "normal" to go to work scared for your health, your life, and your family's lives? It is our right not be scared at work and our employer (in my case, the gov) needs to put measures in place so we aren't afraid. That is law, not just words.

There were 2 steps to the refusal. Refusing to work to my employer, and refusing to work to the Ministry. If my concerns were addressed in Step 1 - to my employer, there is not need to go to the Ministry. We went to the Ministry.

Involved in my refusal were my administration team, a Health and Safety Inspector from my employer, a Health and Safety Inspector from my union, and later in Step 2, a Ministry of Labour Inspector.

I first stated my specific concerns to my adm. They were so supportive in doing everything in their power to address my concerns. I want to thank them for that. They took my concerns seriously. My concerns were all tied to the increased contagiousness of the new variants:
SOCIETY

You May Also Like

Anastasis
Anastasis
@OrdersofM
https://t.co/6cRR2B3jBE
Viruses and other pathogens are often studied as stand-alone entities, despite that, in nature, they mostly live in multispecies associations called biofilms—both externally and within the host.

https://t.co/FBfXhUrH5d


Microorganisms in biofilms are enclosed by an extracellular matrix that confers protection and improves survival. Previous studies have shown that viruses can secondarily colonize preexisting biofilms, and viral biofilms have also been described.


...we raise the perspective that CoVs can persistently infect bats due to their association with biofilm structures. This phenomenon potentially provides an optimal environment for nonpathogenic & well-adapted viruses to interact with the host, as well as for viral recombination.


Biofilms can also enhance virion viability in extracellular environments, such as on fomites and in aquatic sediments, allowing viral persistence and dissemination.
ALL
Vibhu Vashisth
Vibhu Vashisth
@VIBHU_Tweet
AGAMA SHASTRA:THE SCIENCE BEHIND THE MARVELS OF HINDU ARCHITECTURE

Agama Shastra is a Sanskrit word that means the manual for worship, temple building, rituals etc in Hindu Tradition.From Sanskrit,agama means handed down by tradition and Shastra means a commentary or treatise.


Therefore, Agama Shastra is a collection of ideas that drafts the rules for worship, temple construction, spirituality and rituals. It has been a guideline for many Hindus from ancient times. It's a collection of Sanskrit, Tamil and Grantha scriptures that mostly contains...


...methods of temple construction, idol creation, philosophical doctrines and meditative practices. It came into being after years of assimiliation and from a variety of sources. Some parts of it are pre-vedic while some are post-vedic.


Hindu Temples are not just temples, they are the pride and identity of Hindus. They are the symbols of Hindu unity and their religious aspirations. Temples are the places of spiritual healing and are our connection with the creator almighty.


Temples are the priceless treasure of our History, Heritage ,Art & Architecture. Architecture that mesmerises and spell bounds you. And Agama Shastra is that very text which is responsible for such a mesmerising and spell binding quality and beauty of our Hindu Temples.
ALL
Advertisement
Will Buxton
Will Buxton
@wbuxtonofficial
The Haas / Force India argument is fascinating, and the decision of the stewards may still have consequences. The first key finding is that the stewards ruled Racing Point Force India to be a valid constructor. That means that it builds its own cars.

Haas had argued that the cars had been designed by Force India, but as Haas knows only too well, "outsourcing" is legal. The stewards decided that "there is no regulatory support for the argument that Outsourcing of Listed Parts cannot come from a former or excluded team."

In determining this, the stewards also came to the following conclusion: "In relation to the submission by the Racing Point Force India F1 Team that it is not a new team, the Stewards decide that the Racing Point Force India F1 Team is indeed a new team."

This will have a big knock on effect for Haas in its continued arguments over the rights of RPFI as relates to its eligibility and rights under the complex payments structure. Haas has argued all along that RPFI should be treated as a new team and tread the same path that it did.

TLDR? Haas lost the fight but may have won the war. In having its argument that Racing Point hadn't designed their own car thrown out, they gained clarification that RPFI was a new team.
SPORT
Sumit Grrg
Sumit Grrg
@sumitgrrg
These 10 threads will teach you more than reading 100 books

Five billionaires share their top lessons on startups, life and entrepreneurship (1/10)


10 competitive advantages that will trump talent (2/10)


Some harsh truths you probably don’t want to hear (3/10)


10 significant lies you’re told about the world (4/10)
ALL , MOTIVATION , BOOKS
Max Fagin
Max Fagin
@MaxFagin
November is here, and that means a massive shift is coming. And by "massive" I am of course referring to the redefinition of the kilogram unit of mass that the world has been building up to for more than 100 years. Let me explain:


1/ I've had an unhealthy fascination with metrology (the study of measurement) ever since my 2nd year as a physics major when I took a class devoted to duplicating historic physics experiments, so please indulge me for going into heavy detail (get it?) about the kilogram.

2/ So what actually *defines* a unit of measurement? If you're American, you probably know a mile is 5280 feet and a foot is 12 inches and an inch is 2.54 centimeters etc. But where does this chain of definitions end? Is it turtles all the way down?

3/ It's actually not! For all units (even the imperial units used in America) the answers all end with the Système International (SI) unit definitions established and maintained for over 100 years by the Bureau International des Poids et Mesures (BIMP) in France.

4/ At the base of this tower are the SI base units. Just 7 SI base units define every other unit in existence. They are:

Kilogram, kg (mass)
Meter, m (distance)
Second, s (time)
Kelvin, K (temp)
Ampere, A (electric current)
Candela, cd (luminous intensity)
Mole, mol (quantity)
SCIENCE