Important paper from Google on large batch optimization. They do impressively careful experiments measuring # iterations needed to achieve target validation error at various batch sizes. The main "surprise" is the lack of surprises. [thread]

https://t.co/7QIx5CFdfJ

The paper is a good example of lots of elements of good experimental design. They validate their metric by showing lots of variants give consistent results. They tune hyperparamters separately for each condition, check that optimum isn't at the endpoints, and measure sensitivity.
They have separate experiments where the hold fixed # iterations and # epochs, which (as they explain) measure very different things. They avoid confounds, such as batch norm's artificial dependence between batch size and regularization strength.
When the experiments are done carefully enough, the results are remarkably consistent between different datasets and architectures. Qualitatively, MNIST behaves just like ImageNet.
Importantly, they don't find any evidence for a "sharp/flat optima" effect whereby better optimization leads to worse final results. They have a good discussion of experimental artifacts/confounds in past papers where such effects were reported.
The time-to-target-validation is explained purely by optimization considerations. There's a regime where variance dominates, and you get linear speedups w/ batch size. Then there's a regime where curvature dominates and larger batches don't help. As theory would predict.
Incidentally, this paper must have been absurdly expensive, even by Google's standards. Doing careful empirical work on optimizers requires many, many runs of the algorithm. (I think surprising phenomena on ImageNet are often due to the difficulty of running proper experiments.)

More from Machine learning

Thanks for this incredibly helpful analysis @dgurdasani1

Two questions. 1/ Does this summarise the AZ published data :
The plan is to extend the time interval for all age groups despite it being largely untested on the over 55yrs, although the full data is not yet published


Do we have the actual numbers of over 55yr olds given a 2nd dose at c12 weeks and the accompanying efficacy data?

Not to mention the efficacy data of the full first dose over that same period?

I’d quite like to know whether I am to be a guinea pig & the ongoing risks to manage

You attached photos of excerpts from a paper. Could you attach the link?

Re Pfizer. As I understand it the most efficacious interval for dosing was investigated at the start of the trial.


Here’s the link to the

I’ve got to say that this way of making and announcing decisions is not inspiring confidence in me and I am very pro vaccination as a matter of principle, not least because my brother caught polio before vaccinations available.

You May Also Like

Following @BAUDEGS I have experienced hateful and propagandist tweets time after time. I have been shocked that an academic community would be so reckless with their publications. So I did some research.
The question is:
Is this an official account for Bahcesehir Uni (Bau)?


Bahcesehir Uni, BAU has an official website
https://t.co/ztzX6uj34V which links to their social media, leading to their Twitter account @Bahcesehir

BAU’s official Twitter account


BAU has many departments, which all have separate accounts. Nowhere among them did I find @BAUDEGS
@BAUOrganization @ApplyBAU @adayBAU @BAUAlumniCenter @bahcesehirfbe @baufens @CyprusBau @bauiisbf @bauglobal @bahcesehirebe @BAUintBatumi @BAUiletisim @BAUSaglik @bauebf @TIPBAU

Nowhere among them was @BAUDEGS to find
"I lied about my basic beliefs in order to keep a prestigious job. Now that it will be zero-cost to me, I have a few things to say."


We know that elite institutions like the one Flier was in (partial) charge of rely on irrelevant status markers like private school education, whiteness, legacy, and ability to charm an old white guy at an interview.

Harvard's discriminatory policies are becoming increasingly well known, across the political spectrum (see, e.g., the recent lawsuit on discrimination against East Asian applications.)

It's refreshing to hear a senior administrator admits to personally opposing policies that attempt to remedy these basic flaws. These are flaws that harm his institution's ability to do cutting-edge research and to serve the public.

Harvard is being eclipsed by institutions that have different ideas about how to run a 21st Century institution. Stanford, for one; the UC system; the "public Ivys".