Barry Zuckerkorn

This isn’t my field, and some undergraduate philosophy classes I took more than 20 years ago might not be leaving me well equipped to understand this paper. So I’ll admit I’m probably out of my element, and want to understand.

That being said, I’m not reading this paper with your interpretation.

This is exactly what they’ve proven. They found that if you can solve AI-by-Learning in polynomial time, you can also solve random-vs-chance (or whatever it was called) in a tractable time, which is a known NP-Hard problem. Ergo, the current learning techniques which are tractable will never result in AGI, and any technique that could must necessarily be considerably slower (otherwise you can use the exact same proof presented in the paper again).

But they’ve defined the AI-by-Learning problem in a specific way (here’s the informal definition):

Given: A way of sampling from a distribution D.

Task: Find an algorithm A (i.e., ‘an AI’) that, when run for different possible situations as input, outputs behaviours that are human-like (i.e., approximately like D for some meaning of ‘approximate’).

I read this definition of the problem to be defined by needing to sample from D, that is, to “learn.”

The explicit point is to show that it doesn’t matter if you use LLMs or RNNs or whatever; it will never be able to turn into a true AGI

But the caveat I’m reading, implicit in the paper’s definition of the AI-by-Learning problem, is that it’s about an entire class of methods, of learning from a perfect sample of intelligent outputs to itself be able to mimic intelligent outputs.

General Intelligence has a set definition that the paper’s authors stick with. It’s not as simple as “it’s a human-like intelligence” or something that merely approximates it.

The paper defines it:

Specifically, in our formalisation of AI-by-Learning, we will make the simplifying assumption that there is a finite set of possible behaviours and that for each situation s there is a fixed number of behaviours Bs that humans may display in situation s.

It’s just defining an approximation of human behavior, and saying that achieving that formalized approximation is intractable, using inferences from training data. So I’m still seeing the definition of human-like behavior, which would by definition be satisfied by human behavior. So that’s the circular reasoning here, and whether human behavior fits another definition of AGI doesn’t actually affect the proof here. They’re proving that learning to be human-like is intractable, not that achieving AGI is itself intractable.

I think it’s an important distinction, if I’m reading it correctly. But if I’m not, I’m also happy to be proven wrong.

I can’t think of a scenario where we’ve improved something so much that there’s just absolutely nothing we could improve on further.

Progress itself isn’t inevitable. Just because it’s possible doesn’t mean that we’ll get there, because the history of human development shows that societies can and do stall, reverse, etc.

And even if all human societies tends towards progress, it could still hit dead ends and stop there. Conceptually, it’s like climbing a mountain through the algorithm of “if there is a higher elevation near you, go towards that, and avoid stepping downward in elevation.” Eventually that algorithm brings you to a local peak. But the local peak might not be the highest point on the mountain, and while it is theoretically possible to have gotten to the other true peak from the beginning, the person who is insistent on never stepping downward is now stuck. Or, it’s possible to get to the true peak but it requires climbing downward for a time and climbing up past elevations we’ve already been to, on paths we hadn’t been on. One can imagine a society that refuses to step downward, breaking the inevitability of progress.

This paper identifies a specific dead end and advocates against hoping for general AI through computational training. It is, in effect, arguing that even though we can still see plenty of places that are higher elevation than where we are standing, we’re headed towards a dead end, and should climb back down. I suspect that not a lot of the actual climbers will heed that advice.

That’s assuming that we are a general intelligence.

But it’s easy to just define general intelligence as something approximating what humans already do. The paper itself only analyzed whether it was feasible to have a computational system that produces outputs approximately similar to humans, whatever that is.

True, they’ve only calculated it’d take perhaps millions of years.

No, you’re missing my point, at least how I read the paper. They’re saying that the method of using training data to computationally develop a neural network is a conceptual dead end. Throwing more resources at the NP-hard problem isn’t going to solve it.

What they didn’t prove, at least by my reading of this paper, is that achieving general intelligence itself is an NP-hard problem. It’s just that this particular method of inferential training, what they call “AI-by-Learning,” is an NP-hard computational problem.

Though a superhero, Bruce Schneier disdains the use of a mask or secret identity as ‘security through obscurity’.

source

The paper’s scope is to prove that AI cannot feasibly be trained, using training data and learning algorithms, into something that approximates human cognition.

The limits of that finding are important here: it’s not that creating an AGI is impossible, it’s just that however it will be made, it will need to be made some other way, not by training alone.

Our squishy brains (or perhaps more accurately, our nervous systems contained within a biochemical organism influenced by a microbiome) arose out of evolutionary selection algorithms, so general intelligence is clearly possible.

So it may still be the case that AGI via computation alone is possible, and that creating such an AGI will not require solution of an NP-hard problem. But this paper closes one potential pathway that many believe is a viable pathway (if the paper’s proof is actually correct, I definitely am not the person to make that evaluation). That doesn’t mean they’ve proven there’s no pathway at all.

Yes but they only performed the training on the posts and images set to be globally publicly accessible by anyone. In a sense, they took the public permissions as an indicator that they could use that data for more than just providing the bare social media service.

Isn’t the opt-out option to just not make the photos/posts globally public?

It’s a bill to create technical standards by which anyone can mark their digital files with a rough analogue of a robots.txt that says “don’t train on this file,” and a requirement for AI training to obey that standard. It’s for everyone, because copyright is for everyone who creates pretty much anything.

Plus the net worths of Trump’s cabinet is basically the highest in history:

• Betsy Devos (Education): $2 billion
• Wilbur Ross (Commerce): $600 million
• Steve Mnuchin (Treasury): $400 million

Linda McMahon is worth $3 billion but was “only” SBA administrator, not a cabinet secretary.

Jared Kushner is from a family whose net worth is in the billions between all of them, and he got his dad a pardon.

Let’s not forget major donors like the Adelsons, the Kochs, and everyone else.

Yes, Democrats have some billionaires on their side, but the balance isn’t even close.

In my opinion, it’s quite similar to Brexit: maybe you can get a majority coalition to disapprove of the status quo, but good luck getting them to actually propose a more popular alternative. Much less proposing an actual procedure for getting that alternative onto ballots.

Structurally and functionally, our political systems are not set up to run anyone other than the person who won the primary. Changing a presumptive nominee this late in the cycle is fraught with potential complications, but can be done if there’s sufficient support for a specific alternative candidate. Realistically, it’s Biden or it’s Harris. There’s no feasible way to get someone else at the top of the ticket.

Why they went into this election season without even attempting to run anybody aside from Biden I’ll never know.

When has either party ever run a credible candidate against their own incumbent?

It’s not just about pledged delegates

The leadership of the DNC, DCCC, DSCC, etc., are chosen by election, by members of each committee. State parties send their delegates to participate in these things.

despite not being an incumbent

Yeah, that’s what I’m talking about. These are processes that longtime party members participate in, and run on, about the structural rules and procedures to follow, and they’re open to everyone. Elections often pit “establishment”/“insider” candidates against “insurgent”/“outsider” candidates, and there are examples of each kind (or hybrid candidates) winning the nomination in the modern primary system.

It’s more of a spurious correlation: incumbency doesn’t buy the advantage in the nomination race, but reflects that a candidate has the network and resources to have the popular support of their own party. That’s why incumbents always win the nomination, and tend to win reelection in the general.

Tell me, during an incumbent primary, who controls the DNC?

Same as during a non-incumbent primary. The person who won the most recent nomination tends to have an outsized voice in the selection of party officials (because it’s their pledged delegates who vote on all the other stuff). Yes, that means Biden-affiliated insiders had an inside track in 2020, but that’s also true of Clinton allies in 2016, Obama allies in 2012, Obama allies in 2008, and Kerry allies in 2004.

More than a year ago, the DNC adopted new rules—including a primary calendar that ignored state law in Iowa and New Hampshire and eliminated any primary debates—designed to ensure that Biden’s coronation would proceed untroubled by opposition from any credible Democrat.

Which of those changes in the rules do you think were designed to benefit Biden specifically? De-emphasizing the role of Iowa and New Hampshire? There’s been people clamoring for that for decades, within the party.

There’s basically no set of rules that will ever create a credible challenge to an incumbent who wants to run for reelection. It’s a popularity problem, not a structural problem.

No one deserves to be a president any more than anyone else, and treating an incumbent as though they do, without having to go through an open, democratic primary process, is to treat them as more deserving of future authority than other citizens.

There was a primary, and Biden got the most votes/delegates under the rules. Nobody is saying that incumbents should automatically get renomination. Or even that the incumbent should get some sort of rules advantage (like say, the way the defending world champ in chess gets an auto-bid to defend his title against a challenger who has to win a tournament to get there).

The rules are already set up to where any challenger has an equal structural change of winning the primary. They just won’t have the actual popular support. You know, the core principles of democratic elections.

This shifts balance from unelected officials who can be fired by the president (administrative officials) towards unelected officials who can’t be (judges). It doesn’t actually reduce regulatory power, just puts that regulatory power under the supervision and review of even less accountable officials.

Rage isn’t sustainable. Enthusiasm is down across the board, and the question will be whether enthusiasm among left-leaning voters has wanted waned to the same degree as among Trump’s base.

Your description of a drink that takes the world by storm, increasing in market share but dropping in quality may be roughly accurate analogy for a lot of consumer goods, but even in this telling the market is improving if that drink is displacing even lower-quality competition.

In terms of non-alcoholic drinks sold in coolers in convenience stores and grocery stores, we’ve seen the steady march of improving products as an average across the shelves, even if the same product name might be getting worse. In the 80’s, the dominant market share for orange juice in grocery stores was frozen cans to be mixed with water at home. But Tropicana and Florida Natural and a few other brands made a splash with not-from-concentrate orange juice. Old brands like Minute Maid got in on the action, and new brands like Simply rose up, too.

Now, it might be that these brands have gotten cheap with stuff since dominating market share. But if you look at who they took that market share from, it’s unquestionably a lower quality product they’ve displaced.

Across the beverage industry as a whole, you’ve got a whole bunch of newer higher priced drinks, where the unfathomably expensive for 2000 Red Bull is basically the middle of the pack for energy drinks, and where there are so many beverages that cost several times as much as Coca Cola.

So that’s a story of a forward march in higher prices for qualitatively preferred items, over that amount of time. This story I do think applies to processed food and drink, as well as electronics, prepared food, home furnishings, and cars. We expect a lot higher quality every year, as the things get more expensive, and we feel annoyed that any particular brand or model seems to be slipping in quality while we as a consumer market tend to move up the chain.

We’re angry that streaming seems to be slipping back to cable-like quality, when streaming as of 2024 is still a much better value proposition than cable in 2014. The displacement is happening in two directions, for a net benefit to the consumer in a way that doesn’t feel like a benefit. Same with music, video games, etc.

The real story is that housing, education, healthcare, and dependent care (both childcare and elder care) have gone up so much faster than inflation that these things are finally squeezing normal people out of their comfort zones right when the other stuff stopped dropping in price as much as before.

The non-cynical answer is that they’re counting contractor/vendor time in this full time equivalent answer. Which would probably be a good thing, because I imagine that the best people in cybersecurity aren’t actually employees of Microsoft.

To put it in more simple terms:

When Alice chats with Bob, Alice can’t control whether Bob feeds the conversation into a training data set to set parameters that have the effect of mimicking Alice.

Your comment missed the mark entirely.

Not sure why you’re saying that. I wasn’t disagreeing with any of your points, but adding to them another angle that answered the parent comment’s concerns about whether leaving wifi on for airplane mode drains battery. You addressed the cellular radio side, and I was adding a separate point about the WiFi radio that complements what you were saying.