I think it's great to talk about other methods of training or architectures that don't require so many parameters. The point about how BERT consumes vastly more text than humans do when they learn to read is interesting. But trying to phrase this like an environmental issue just seems disingenuous and misleading.

TFA:

"What does this mean for the future of AI research? Things may not be as bleak as they look. The cost of training might come down as more efficient training methods are invented. Similarly, while data center energy use was predicted to explode in recent years, this has not happened due to improvements in data center efficiency, more efficient hardware and cooling."

Also. One flight in and of itself isn't an environemntal problem; thousands of flights are.

Training this one instance of a model isn't an environmental problem; I would be curious to see some educated guesses about the number of models being trained over the next ten years. Not an expert but I know that use of ML is exploding - lots of new use-cases and thus lots of new models.

So it makes sense to me think about this stuff.

To

There are hundreds of thousands of flights per day. Adding an additional flight, or even an additional thousand flights, to substantially improve Google doesn't seem like a big cost. Consider: Would your life be worse if one random trans-American flight was cancelled today, or if Google searches became 10% worse?

Another way of thinking about this same point, is, why is the author writing about language models if they are so concerned about the environment. Surely the airline industry is a better subject as, again, they fly hundreds of thousands of flights each day. It's hard to take someone seriously when they are focusing on an infinitesimal part of a huge problem.

It's also misleading because there is a difference between energy used by an airline flight, where the energy comes from burning jet fuel, and energy used in a data center. In Google's case (the people training BERT) the energy used was 100% renewable - Google reached that goal in 2017. Perhaps Open AI didn't use renewable energy to train GPT-3, but I wager they didn't power their machines by burning jet fuel either.

Maybe electricity used to power to train language models will become a meaningful issue at some point in the future. I don't think that future is close at hand though.

That's a good point about renewable energy sources.

I'd still argue it's reasonable to assume that ML will consume more power as use-cases for it grow (i.e. more and more models are trained), and, therefore, I don't think it's unreasonable to consider the energy usage.

Honestly - I think what you may have heard is "ML is bad/evil." And yeah some folks probably feel/vibe that.

That's different from "Like any other technology the full context of ML deserves to be considered, and since it's so hyped by billion-dollar companies right now, maybe a bit of pushback on all the hype isn't a terrible idea."

And yeah we should definitely look at the airline industry; pretty sure Mother Jones does, and there's no reason we can't look at the energy impact of multiple industries.

Likewise, yes, language models use energy, but objectively, it's not that much energy and the sources for the energy are sometimes green and always efficient (i.e. Coming from the grid and not by burning jet fuel).

You can't really solve the problem of lung cancer by addressing scented candles. Reducing energy used by language models likewise won't have an effect.

https://blog.google/inside-google/infrastructure/data-center...

The more folks who elect to pick where to compute based on low electrical carbon intensity, the faster the grid turns over to clean generation. You must vote with your fiat. I encourage technologists to include this consideration in their workload scheduling requirements. Renewables are almost always cheaper than fossil generation as well.

Meanwhile, the aforementioned transatlantic flight cannot get greener, whether Bert is trained or not.

The mentioned paper itself notes a 300,000x increase in compute used for training language models over the last 6 years.

I think the point is, if nothing changes, the costs will be very significant very soon.

The energy efficiency of machine learning hardware is progressing at a rapid rate. It's not accurate to assume that the energy costs will stay the same. Just look at how much it would have taken to train something like GPT-3 five years ago.

ML model size has had it's own Moore's Law, with standard model growing exponentially in size [1]. And this implies model are going butt up against the limits even more than they have already. Whether the "bitter lesson"[2] of ML is true inherently is up for a question. That current researchers have accepted it seems given.

[1] https://openai.com/blog/ai-and-compute/

[2] http://www.incompleteideas.net/IncIdeas/BitterLesson.html

2019 was peak flight with 38.9 million flights over the year. That averages to 106k flights per day.

You make good points, but this one isn’t quite what I meant. I didn’t mean we are trained by evolution for a particular language, but that evolution selected for a language skill. In other words, we are “pre-baked” with the ability to learn a language. It would be analogous to a DL model being trained for generalized regression but not a particular problem. To that extent, I think we’re saying the same thing. Some of the theories related to our generalized learning abilities postulate they stem from this base ability (our aptitude for music, for example, being a consequence of our language learning ability)

>I see BERT as non sensical. You need to be scientific and have a mathematical theory

This is a matter of contention. A lot of science progresses by starting with an empirical result that drives a change in theory rather than the other way around. I can’t remember who to attribute it to, but there’s a quote to the effect that “‘Thats odd’ is the most productive phrase in science, rather than ‘Eureka!’”

Yes it did, and we do know a lot about general principles behind, from different disciplines. This is still an early science. NLP research still hasn’t considered many important aspects of language learning that we have discovered in such a short period of time.

> starting with an empirical result

What makes you think these benchmarks are empirical? They were hand constructed to fit some objective, assuming that being good at the said objective is required for NLP tasks. Where’s the empirical experiments to validate the notion that said objectives lead to language? Science hasn’t worked this way, datasets aren’t constructed, you do an experiment and MEASURE it. Then you make models, try to explain the phenomenon, and test new ideas and validate your models. Can your model extrapolate new information and suggest new experiments to validate? I used the word extrapolate over predict intentionally.

I assume the empirical results are the validation sets run. I.e., the tests that show it provides better results than the base rate. Again, it’s important not to conflate verifiable results with understanding the underpinnings of why it works. If you’re walking and I race you with my car over and over again, I can conclude my car is a faster mode off transportation without understanding anything other than “push right peddle to go faster”. My ignorance doesn’t invalidate the results

And it's not the whole picture to say the brain only uses 15 watts - when there's all sorts of necessary support systems that it couldn't run without. So it's closer to 100 watts (2000kcal/day)

All in you require about 100W (2000kcal/day), maybe 3 times that when doing a lot of physical activity. Boston Dynamic's Spot uses about 400W. I can probably outperform it in some disciplines while it would beat me in some others. That would be a fair comparison.

You’re making sweeping assertions that require domain experts in several different disciplines. The scientific evidence points to the contrary. It is demonstrated that humans have a general ability to learn wide range of things, without being genetically programmed to. None of us evolved to drive cars. Yet nearly everyone in my grandparents generation were able to learn this totally new skill despite being a new invention, where you couldn’t possibly have had time for evolution to act. They weren’t genetically evolved to drive, it was learned within their lifetime and generation. There are tribal humans in different parts of the world that haven’t developed written language. Yet you can teach them written language. Where’s your “pre trained brain” theory there?

That is if we don't kill ourselves by making stupid mistakes.

I think 30 years is probably reasonable, if for “30 years” one reads “twice as long from now as commercially viable fusion generation actually was when first widely hailed as 15 years away”.

Theory is required for understanding, not for discovery. They are two different things. What gets a lot of scientists and statisticians worked up is how ML can outperform traditional models without always being interpretable. It’s like claiming the Wright brothers couldn’t build a flying machine without having a thorough understanding of the mechanics of flight. I can improve performance of my car by remapping the fuel without necessarily understanding the nuances of optimizing enthalpy. There’s levels of understanding, and they may not correlate completely to effectiveness. To that extent, it’s more engineering than science

See also: Sorites fallacy.

There is a limited desire for flights, which benefit people as far as it allows them to get things where they need to be, and no more than that; there is presumably an unlimited desire for machine learning.

The human brain is the output of an incredible number of generations of training, representing a vast consumption of energy. Most of the learning that informs the brain happened before this hypothetical five-year-old was even born.

But the actual learning process of a single brain uses very little training examples and is very energy efficient. A brain that uses only 20 watts of power.

And no, I don't ridicule AI ethics researchers: the energy cost of AI is negligible to the serious issues. The real harm stems from other areas.

From the paper they cite

    algorithm    kWh-PUE
    ----------   -------
    ELMo             275
    BERT(base)     1,507 
    NAS          656,347  

More importantly, a human brain isn't trained from scratch at birth. It's had a few hundred million years of pretraining.

If we want to account for power supply, temperature control etc. we could use the entire calorie count of a human over 35 years, leaving us with around 30000kWh (assuming a health, not-overweight human). You could now argue that that's too high (the human is doing a lot of other things). But as you pointed out it's a bit of a pointless comparison anyways as humans don't start from scratch.

That's still too low, considering you can't just have a human eating food and educating themselves in a vacuum.

We need to account for energy going to and from school, shelter, heating, clothing. Basically all the necessities that enable someone to learn (and their professors)

That's another 5x at a minimum.

This doesn't take you anywhere.

Basically, one should really not discount human ability, especially in tasks considered mundane. Such skills often are often considered "mediocre" not because they're actually easy but because all human can do them, see:

https://en.wikipedia.org/wiki/Moravec%27s_paradox

No there are no risks of carbon footprint. Energy is cheap. Just build nuclear power plants.

Or just build solar and wind and train models half of the time if one is paranoid about nuclear power plants safety.

They need to mention that the total number of people training large transformers from scratch is very very small. If wager that the total number of different, uniquely trained (not using previous weights - which reduces compute necessity by massive amounts) language models in existence is in the low hundreds

I'd claim that these mass language models serving as the underlying encoding backbone behind more specific systems actually save energy and compute compared to the previous methods (needing far more data and thus more energy spent on getting it combined with less efficient representations like tf-idf causing many classifers to perform very slowly and thus burn lots of energy)

Also, much of the recent research in this field is about model pruning, quantization, and any technique you can imagine to reduce training and inference time or memory requirements.

All in all, big language models are a net positive for the environment. The effeciency gains in any number of fields from increasingly sophisticated NLP systems far, far outweighs the costs or of training them. Foundational research in environmental conservation will be accelerated by effective NLP semantic search and question answering systems. That's a single, tiny example of the potential for benefits from large language models.

Pick a better target.

One can also observe that humans have largest, among the animals of planet Earth, share of body energy consumed by brain and the future humans would probably have even higher share. In technology we observe the same - the pinnacle of technology - CPU - have practically 0 thermodynamic efficiency and the share of energy consumed by computers grows, and i think it will be only growing. Intelligence eats the world.

I suppose there were a lot of arguments about usability and various constrains/impossibilities 30+ years ago when cell phones only appeared at $1+/minute and were available only at a very few places. Yet here we are. The cheap access to space will do the same for various tech-in-space (how about unlimited access to space with fineprinted "cap of 1000kg/month" :).

No way they'll allow that to happen I think.

There's of course also

- deflation meaning anyone who believes in Bitcoin and uses one to buy something is either desperate or a fool

- as well as practical issues like energy consumption

If you visit a foreign country how do you know to trust whether to trust a specific bank? You probably look at cues like whether it occupies an expensive skyscraper in the center of town? Do you see its ads around town? Does it sponsor the local soccer team? All credible signals that are hard to fake for a fly-by-night scammer.

All Bitcoin did was formalize this process. At any given time there are many chains that all purport to be the canonical history. How do you decide which one is authentic? By looking at hard-to-fake signals. In this case the accumulated hashing power behind the chain. Looking for whoever spent the most hash work is fundamentally no different than checking to see which bankers are wearing the most expensive suits.

Any system with trusted intermediaries will waste resources on costly signaling. The only question is whether crypto mining is more of a fundamental waste than traditional signals, like high-paid bankers and prestige real estate.

Bitcoin uses 600+ kWh per transaction currently. With current use, to process VISA's 1700 transactions per second would consume >36 PWh annually. That's significantly higher than global electricity consumption and capacity. At current US electricity prices it would be significantly over 4.5 trillion USD annually. Global annual banking revenue is under 6 trillion: https://www.mckinsey.com/industries/financial-services/our-i...

> The only question is whether crypto mining is more of a fundamental waste than traditional signals, like high-paid bankers and prestige real estate.

The answer is yes. Incredibly higher.

For example miners use energy to compete for the mining reward which is awarded every ten minutes. I can think of many ways of modifying the network to increase the rate at which transactions can be processed that do not increase this mining reward. (I do not understand why none of those ways have been adopted.)

One of the primary reasons that reward is not lower is probably because the amount was set when Bitcoin was created, and people worry that changing it would set a precedent making other changes easier.

Lighting attempts to decrease this load by effectively batching transactions, but its not a huge difference compared to the orders of magnitude differences that exist already.

the vast majority of bitcoin/crypto mining uses renewable or otherwise wasted energy and this is the only economical way to do it! you have been intentionally misinformed by the amount of energy used and not the source of energy used. It is completely a red herring to just read a headline about how little electricity a tiny country uses and that mining uses the same amount.

what? "wasted energy"? yeah a lot of energy is lost as it cannot be transported to commercial and residential areas economically. so miners set up processing at the source of the energy and use it. in fact, a lot of it actually reduces pollution creating the polar opposite of what you believe, in those circumstances it is a sustainability solution.

now aside from fighting me about your worldview, this is also an area to remain vigilant about! nation states can absolutely mine at a loss when they turn to competing for control of the cryptocurrency networks for geopolitical reasons. right now it is renewable.

but what about the hardware, the single purpose hardware and e-waste? there is a large trade of "obsolete" hardware, as it is economical to use for people with the cheapest power. this is also another specific area to be vigilant about, by making sure the infrastructure is in place to put that hardware and keep it active, instead of landfills.

B$ still looks ridiculous, let me explain.

* world electrical energy consumption: ~25000 TWh [1] * bitcoin energy consumption index: ~77 TWh [2]

Bitcoin today consumes as much as 0.3% of the total energy available on this planet.

That's fine, if half the world would use it and would do meaningful things with it.

What is it actually used for most visibly?

As a betting ground with galactic momentum, a technology promising to be the solution to everything, and just outrageous claims that only the bovine left to be excited about.

It's not that the algorithms and data structures are not cool, the certainly are - but we can do so much more today with technology than this.

[1] https://www.vgb.org/en/data_powergeneration.html?dfid=98054 [2] https://digiconomist.net/bitcoin-energy-consumption

The amount of energy without detailing the source of energy is not a valid argument to support any argument you have about why it is being used. Move off of it, or criticize the sources in a more nuanced educated argument. It should be 100% renewable or part of a sustainable solution, not only will you get further in your argument by focusing on the market participants that actually are wasting industry, you will also be helping!

Is that "educated" enough for you?

Now, when electricity is generated, you can transmit it over those power grids, even to places that aren't right next to the dam. You can even store a limited amount for later use.

Ever kWh that gets redirected away from frivolities like BTC can go toward something that would otherwise be powered by fossil fuels.

I thought you were "educated" here? Please do keep up.

A) from some sources: Energy that wasn't going to be sent over those power lines

B) from other sources: a hedge against the loss

C) in some sources a combination of both A and B

If you get around to noticing, I have refrained from any snark with you and I wonder how long you will keep that up

Regarding A), there's no particular reason why that electricity can't be sent across power lines.

B) So what? Send it anyway, and replace 85-92% of the energy use.

Now let's talk about your unique points than the original person I replied to: places have had decades to send more power over power lines and they didn't. You tell me why that is. I assume there were financial reasons and related impracticalities. But admittedly I have never asked and only react to the reality that energy producers have been receptive to the additional use of their energy for the aforementioned reasons I listed. Their output hasn't changed and to them it is a more efficient use of it. Are they lying? Are they ignorant of alternatives? Just lazy even though their laziness would therefore predate crypto mining?

Either way mining on premise is an immediately applicable economic incentive that simply got you to notice that you didn't like it.

Best case scenario then is that it gets you into action to implement a solution nobody else noticed was applicable. Society might be getting somewhere because of you, that's so exciting.

Nonetheless, financial difficulties are not relevant. What is relevant is reducing the world's carbon footprint, and that needs to happen ASAP. I, personally, think it's more "exciting" that human civilization might survive another century than to keep track of the solution to some useless, financialized math problem. But, that's just me, I suppose.

Corroborate that with a news source that you happen to like.

Like I mentioned renewable or otherwise wasted energy. This is one of the otherwise wasted energy examples that is a sustainability solution right now, in comparison to wishful thinking.

Likewise, if you think fossil fuels will be scarce in the future, you can be bullish on the price of gas, but not bullish on things based on gas.

The best AI will always be a meaty human.

That is... certainly one way of putting things.

If even then you still want to argue, aluminum consumes 17,000kWh per ton ton which is 11 BERTs. I don't see weird criticism about mining consuming energy. Doing stuff takes energy and while the sentiment against ML seems to be "it's snake oil", it is still very real research with actual real-world impacts.

Finally, we are in a transition phase where ML is done on GPUs which consume more energy than dedicated ASICs. We already have Google TPUs and Amazon Inferentia which can be used today. Power consumption will go down as dedicated hardware gets better and better.