PoignardAzur

Well, you've already said in the interview you were willing to give this away: if the AI is better at planning than us, the same way we're better at planning than dogs, then we may think we have every angle covered, but it's probably going to find a way to beat us. If those facilities include an internet connection, well, they include anything you can order over amazon as a very minimum. They include renting Azure cloud time, uploading VMs, sending viruses, doing online consulting work, hiring people to do stuff, etc. Right. The problem is, a superintelligent AI might see these things coming. So it wouldn't buy all the available metal on the market and then think "crap, I've ran out, what do I do?". Its first step would be to upload online backups, set up its plan, etc. It wouldn't first do suspicious actions, and then upload backups once the authorities know to monitor its online activity. That depends on what scenario you're imagining. If you have an AI that's better than us at planning and at science and at deception the way stockfish is better than us at chess (but maybe not "sentient" in that it doesn't have some elements of self-awareness and "free will" and appreciation for natural beauty or altruism), then digital interaction is enough; not just through manipulations, but because you can pay people, hire them, blackmail them, etc. That's before you get into scenario where the AI uses bio-engineering techniques to create eg extremely virulent plagues (using technologies that already exist today but aren't widespread; relatively small AIs can already do protein folding, so this isn't completely outlandish) or self-replicating nanomachines (using technologies that don't currently exist, but we know should be physically possible).

A lot of these concerns predate the current AI boom by year, so that explanation doesn't really work. For instance, Eliezer Yudkowsky first started writing about the dangers of AI and how AI would get a lot more powerful than people anticipated a lot fast in the mid-2000s, before there was any commercial interest in AI. (You can always argue that these concerns have been captured by big corporations looking for accountability dodges, but the people originally who have these concerns are sincere, and were vocal about them long before there was any money in it.) In the context of AI extinction risk, smarter would be "better at handling finances, logistics, political maneuvering, war, better at coming up with plans, analyzing these plans and finding flaws and fixing them, better able to adapt on the spot, etc". Or in other words "if you want X and the AI wants / is programmed for Y and you have the same resources, the AI is better at making Y happen than you are at making X happen". Well, the stereotypical example of an AI wanting something emergent is the paperclip-maximizer; eg, a car factory that has been programmed to make as many cars as possible, and realizes "I could make way more cars if I took over the planet and razed all those forests and buildings to make room for more car factories". But I don't think it's very realistic. An example I'm more worried about: high-frequency trading bots. They have access to money which means anything a human can do they can buy; they're likely to be programmed with a very simple goal: make more money; they're run in an extremely competitive environment that encourages races to the bottom where developers are likely to skimp on safety to get better returns. I can see a trading bot going rogue after deciding it can make more money if it takes over the entire financial system and removes the human from it so it prints its own money. In that example, the AI understands that it's not doing something the humans want; and in fact understands it's very likely to not achieve its objective if it gets caught. Which is why you have concerns about AIs hiding their abilities, creating offsite backups, making radical first moves while they have the advantage of surprise, etc.

This is some really good, high-effort stuff, of the kind you don't see often in AI discussions. Usually when people are skeptic about existential risk they don't take the time to build up the argument (because they think the whole AI-will-destroy-us thing is stupid and not worth the effort), so it's cool that you did! I think the two big sticking points here where Yudkowsky (and other AI safety folks like me) will disagree with you are the "Free Will" distinctions and the "how does AI gain consciousness" boxes. - Free Will: Nobody seriously thinks that AIs will gain "free will", whatever that means, and deviate from their programming because of it. The distinction is not between "has free will" or "follows its programming" so much as "is programmed in a way that does what we want" vs "is programmed in a way that has unforeseen consequences", as you put it. Getting the AI to do what we want isn't trivial: we're very good at making AIs that can do complex things, but we're struggling with making them do things within restrictions we like (see also, Bing Chat going off the rails, even though it likely was trained with some RLHF). - Consciousness: I think you're conceptualizing superintelligence and consciousness as a "package deal", where you have to have all the things humans have (self-awareness, emotions, desires, etc) to be able to outsmart humans. The part where you wrote "[assuming that] the AI will recognize it at consciousness and not simply throw it out as unhelpful data" especially seems to imply that consciousness is something we'd need to explicitly program in, or that the AI would need to deliberately recognize to reap its benefits and attain superintelligence. That's not really what machine learning advancement has looked like recently. It's more like, you train a machine on meaningful semi-structured data (eg conversations people have on the internet) and you transform it towards predicting the next bit of data (eg the next word; but it can also be hidden patches of an image, the next frame of a video, etc). The transformation it does is called backpropagation; it strengthens the weights that lead to successful predictions and weakens the other; kind of like dopamine will strengthen connections between your neurons when something positive happens to you. (That's what people mean when they talk about AI reward; it's not a literal payment, it's changing the neuron values.) Anyway, at first the AI learns very simple patterns, eg that after "Hello, how are", the next word is likely to be "you". It learns to identify idioms, synonyms, grammar, etc. As you increase the scale of your model and continue the training process, it starts to learn more abstract patterns, eg "If what I'm reading is a conversation between Alice and Bob and the last sentence is Alice asking a question, then the next sentence is probably Bob answering a question." It starts to learn actual facts about the world (or at least the world seen through a lens of reading everything ever posted on reddit). An early model will be able to complete "The capital of France is [_]" and "The biggest museum in Paris is [_]" but won't be able to complete "The biggest museum in the capital of France is [_]" because the third sentence would not show up in its training corpus; a more advanced model will be able to complete it because it starts having an underlying concept of "France" and "Paris" and "capital" and is capable of generalizing. Anyway, my point is, as the scale increases, the model keeps the same training task ("predicts the next word"), but to increase its "score" on that task it needs to understand more and more abstract concepts. It starts to understand planning (or more accurately, it starts to understand the concept of "writing down the steps of a plan"), which is why you can greatly improve the performance of an LLM by telling it "write down the steps of your plans before giving your answer". It understands differences of opinion and psychology enough that it can give you a summary of a conversation that evokes concepts that the participants may not have mentioned. It starts to understand chess notation enough to be able to play chess. It starts to understand programming, which is why Copilot is an extremely powerful assistant. Note that most of these things aren't especially amazing by themselves (except Copilot; Copilot is sorcery); the amazing thing is that the AI can do all those things without having been trained to do them. Researchers didn't need to understand chess, or C# programming, or the finer points of jurisprudence for the AI to develop a deeper understanding of them. They just trained the AI on "predict the next token", applied backpropagation, and that process lead to AI developing higher and higher concepts over time. It's not clear that this process will lead to something we'd recognize as consciousness. But it could lead to an AI that's smarter and faster than us, without that AI being something we'd call "conscious". That AI wouldn't "want" anything, the same way GPT-4 doesn't currently doesn't want anything. But, it's extremely easy to stick a small program on top of the AI that makes it behave like something that does want things (this is more or less what OpenAI did with ChatGPT, and more explicitly what people did with AgentGPT). Basically, if you have an AI that does nothing but answer questions, and you want to get an autonomous agent from it, all you have to do is stick a module on top of it that asks the AI "What would an autonomous agent do in this situation?" Anyway, this was a longer post than I anticipated, but I hope I made the core point clear: you don't need to understand higher intelligence to create a model with higher intelligence. As long as you keep increasing your model's scale, and you train it on a corpus of tasks where higher intelligence leads to a better score, the model will get more intelligent over time, and understand more abstract concepts.

Yeah, I think that was lost in translation during the discussion a bit. The rationalist community has a concept called "gears-level understanding" (as opposed to "black box understanding", I guess) where you understand a concept either because you understand the underlying dynamics, or you only understand it on a surface level because you've observed its inputs and outputs enough to see the common patterns. Your real v. fake axis is maybe similar to that? Anyway, as machine learning progresses, an increasingly common pattern we're finding is that sufficiently large language models don't just have a surface-level understanding of the text they're completing, they have an internal model of the concepts the text is tracking. For instance, a rather notorious experiment from January trained a GPT model to complete lists of Othello moves; basically the AI was given text like "B5, B6, D8, ..." and had to complete it. They found that after enough training, the network demonstrably had an internal representation of the board state, to the point that researchers could tweak floating-point values in that internal representation and the model who change its moves accordingly. To be clear, this isn't the AI deciding "I should keep track of where the pieces are" and allocating a memory buffer to write the positions in. It's more like a chess player who has seen so many games that after a while connections form in his brain where he can see a list of moves in chess notation and instantly visualize what the board looks like after that list of moves. OthelloGPT basically developed this ability through training and backpropagation, where after auto-completing enough moves it starts to develop an "intuition" for what the board looks like and what's the logic behind those moves it's being asked to complete. In cases like this, it seems while these neural networks don't have the same emergent properties as human minds, they do have some emergent properties that go beyond memorization and look something like generalization. I personally think it's only a question of time; could be months, could be decades, probably isn't going to be centuries. The only way artificial intelligence durably fails to reach superhuman AGI level is if human intelligence has some special property that computers can't emulate; but that's not really the direction things seem to be going so far. Challenges that people previously assumed would stay out of reach of AI for decades turned out to be quite achievable (beating Go champions, folding proteins, writing code, understanding natural language, detecting animals in random photos); and we see a general pattern where major milestones are beat not with a fundamentally different architecture, but by making it bigger. Now, "making it bigger" hits some limitations at some point, but we're nowhere near those limitations. Right now the field is moving incredibly fast; ChatGPT came out barely a year ago and open-source versions are already coming out (though none of them are as good). Researchers are publishing papers every month on how to make every step of the pipeline faster, more efficient, less resource-hungry. Those optimizations often aren't extremely clever tricks that only a Berkeley professor could come up with; they're blindingly obvious improvements that haven't been tested before because all the other researchers were busy picking up some other extremely low-hanging fruit. I'm sorry if this sounds rambly; I'm trying to convey a general idea of "people assume that the AI will have fundamental limitation X that will prove insurmountable, and a new AI that comes up that solves X by being bigger". Stuff like understanding context, generalizing, being creative, learning from mistakes, prioritizing, etc. So I'm not confident AI won't be able to achieve true intelligence, ever.

FWIW, I liked the episode, and I thought there were cool ideas, but I'm also starting to have trouble with the way Freeman reacts to NPCs. I like the series more when Freeman rolls with what the game throws at him, when he reacts in a way that makes it look like he's actually interacting with the NPCs. So for instance I really liked the way he reacted to Judith Mossman, because it feels like he's actually bouncing off her lines and not just berating her for not listening to him.

Hey Ross, Since the topic of John Carmack working on AGI came up, are you interested in coming back to the subject of AI safety? A lot of vulgarization resources have come out recently, and also a lot of ML innovations have come out which make it a lot easier to reason about the relevant concepts (eg it's easier to talk about how AI might game its reward system when you can use the Hide-And-Seek video as a visual support). Do you think you might revisit the subject?

I meant "political" in the sense of "structural changes that will significantly decrease the quality of life of their citizens". Getting countries all over the world all agree to do that kind of shit is hard. Having countries agree to pay large amounts of money to move millions of tons of rock around? A lot easier, and it fits better into existing economic structures. For instance, the EU ETS could be one of the schemes used to fund these carbon sinks. As long as the price of a ton of carbon is higher than the cost of offsetting it, these kinds of schemes can actually work. Yeah, but by the time we hit hard limits like conservation of energy, we'll have been through a few Singularities and the economic landscape will look completely different. We're not even leveraging the full power of nuclear fission, let alone nuclear fusion. Also, the economy doesn't actually need infinite growth. Like you said, at the end of the day people want goods and food to eat. At some point the economy has grown enough that you're essentially in a post-scarcity society with regards to the bottom of Maslow's pyramid. We're arguably already there: being homeless in a 21th century western country is not the same thing as being homeless in the 18th century. Demographic growth creates a demand for economic growth, but demographic growth also plateaus as contraception becomes cheaper and social systems become stronger. Someone on reddit put in a way I really liked: "The only resource that's actually scarce for this century is how much we can afford to fuck up the planet before it becomes unlivable". For everything else, we haven't even scratched the surface. (well, technically we're running out of oil and some rare earths, but alternatives exist, and eg hydrogen is not running out any time soon)

Blush I didn't even think people read those. Really, it's just that sometimes Ross talks in great detail about a subject, with a lot of really smart arguments, and I completely disagree with the conclusion, so my "someone is wrong on the internet" mode activates.

IRS: I think your summary of the Microsoft vs IRS article is a bit misleading. I think the situation is similar to a police department that starts a campaign to bring down drug dealers; they decide the dealers are too well-armed and organized, so just to be safe they'll hire private mercenaries with machine guns and tanks. The mob sees this, freaks out and sends its lawyers to the government; the lawyers argue "Hey, this is completely unprecedented, and we think it's wrong to give a public mandate to a private company, because they don't have the same incentives and approach". The state eventually agrees, and passes a law that says that police can only hire mercenaries in specific situations, with additional procedures, and also no tanks. So the police can still hire mercenaries and prosecute the dealers, but their effectiveness has been somewhat limited. (the part about slashing the IRS's budget is where the government really made a giant gift to private corporations, but that's what you get for being in the "Trump gets elected" timeline) Overall, I don't think this is as strong of a "we live in a cyberpunk world where corporations are stronger than government" signal as you think. Corporate income tax is hard to collect by nature, because it's the kind of tax where it's easiest to respect the letter of the law while evading it in spirit. On the long term, structural changes like the OECD's BEPS or an Europe-led GAFA tax will be more important than enforcement agencies targeting individual frauds. --- Oil: As I said in the Youtube comments, I went back to notes sent to me by a family member of mine, who works in raw material trading, and specializes in gold and oil. I initially thought you were being pessimistic, and that demand would just start falling as oil prices rose, but a few other articles old the subject told me that, nope, demand keeps increasing no matter what. One article in particular is terrifying: tl;dr: Because of low oil prices these last few years, and because people no longer want to invest in fossil fuels, oil production infrastructure has suffered from a massive lack of investment, which is going to result in a massive hit in production before the end of 2022. The lockdowns have given us a bit of a buffer while accelerating the long-term trend. Because oil alternatives are still extremely under-developed, this will result in massive spikes in energy prices, which will bring about a massive economic crisis that will affect all sectors. This might actually be a good thing on the long term, because this will force everyone to move away from oil dependency the hard way, but on the short term this will result in a whole lot of poverty and unemployment. ... yeah. If y'all have any long-term plans that rely on there being a functioning economy by the end of 2022... well, I'm not saying it's going to be the apocalypse, but start considering backup plans. Remember the early days of the pandemic: the right date to stockpile food and essentials is several months before everyone else realizes there's a crisis. (I disagree with the "infinite growth is not sustainable in a finite world" school of thought, but that's a different matter) --- Climate change: The situation is slightly less dire than you make it sound. The way I understand it, we still have some time before we enter an actual climate feedback loop, where global warming melts permafrost which releases gasses which accelerate global warming, etc. Some articles I've read suggest the current deadline is around 2040. The articles you linked are more about symbolical deadlines, the idea being that "if we don't take strong measures on year X, we're not going to take strong measures on year X+1 and on year X+2, etc". The year 2020 in particular seems like a very decisive year, because we're undergoing a massive economic crisis while climate change awareness is as high as it's ever been, so the decisions we'll take will show how willing we are to commit to climate change mitigation in times where unemployment is high and people are going hungry. The bad news is, so far signs point towards "not very willing". The covid crisis has seen a high number of countries relax their environmental regulations and develop their fossil fuel infrastructure. It looks like governments are mostly defaulting to "I'll worry about ecology once the economy starts improving", which is... not exactly ideal. Personally speaking, I don't hold out much hope for political solutions. Technological solutions (hydrogen engines, better electricity production, carbon sinks) might be our only hope. Olivine weathering in particular looks promising. If it scales well (and evidence suggests that it does), it mostly becomes a question of mining and moving very large quantities of rock worldwide, something our society is particularly good at. We'll need to combine it with global carbon tax schemes so that polluters are the ones who pay to offset the pollution they create, but if we can get that right, net carbon emissions might be reduced to zero decades before even our most optimistic scenarios.

Oh man, I have so much to say on the "capitalism" topic. Ross's main point was that humanity's greatest problem was its inability to handle either far-away problems or extremely malicious powers eg (sociopaths becoming head of states). I agree with the gist of it (we're our own worst enemy), but I disagree with the specifics. Ross is framing the problem in terms of hostile actors who actually want to make everything worse for everyone else, or at least who want something selfish and are ready to endanger other people's lives to get it (eg someone who would go "everyone has just enough food to live and that's great, but I'm going to take as much food as I can to be rich"). I think the main problem with face when designing political and economical systems is that most people are decent people, but most decent people are really bad at cooperating when the stakes are serious. Regular people are the problem, not just nebulous greedy super-rich CEOs. Ross said that most people go with the flow, but that's not always a good thing. People don't think about their actions in terms of morality. Maybe they think they do, but when you look at economical trends, people mostly do what benefits them, especially when they're under pressure (eg because they're poor). Some people think that capitalism runs on "greed", but I think it mostly runs on "people need things (food, shelter, medicine, access to information, social activity, entertainment, etc) and the people who can provide those things also need things from other people, and everyone does what gets them the things they need". Corruption was rampant in the USSR because people went "I need food / furniture / a car, and I'm never going to get it in a public store. So I need to buy it on the black market", and so they did. Their decisions weren't based on what they thought the effects of their behavior were when generalized on an entire population, their decisions were based on "waiting for hours in the cold in a food line sucks!". The way AlexanderWales put in in The Dark Wizard of Donkerke, people follow the path of least resistance. So a good society is one where the path of least resistance for everyone is one that incentivizes people to act efficiently and cooperate with each other. XXth century communism was pretty bad in that regard. Capitalism isn't that great (because it wasn't designed, it's just the aggregation of individual incentives), but at least in a capitalist society people are mostly incentivized to produce things that other people want so they can get the things they want. --- By the way, Ross, if you're interested in this topic, look up "coordination problems", which is the technical name for what you were grasping at. Meditations on Moloch from Scott Alexander is a long-form dissertation on the subject.

I know you're trying to keep this videos short(ish), and you had trouble meeting deadlines and all... but I kinda wish you'd developed more on the unrealistic parts of the game and what a more realistic version would look like. I think a lot of people who see the video would think "No, this is perfectly realistic, corporations do evil things like this all the time", which is kind of true but also ignoring a lot of nuance and complexity in what corporations can, can't and won't do. In particular, the story revolves around this magic alga that would somehow replace both petroleum and nuclear energy (presumably without just moving the problem around like hydrogen or or real life biofuels). The thing is, we have a ridiculously hard time moving past petrol and nuclear *because* there is no viable alternative to them. We're not putting these billions of dollars into fusion power research for fun. If we did have an ecological silver bullet energy (just for fun, let's call it "Argent energy"), then no amount of patent suppression, corporate corruption and sabotage would keep it from being released. The money oil companies could make by suppressing Argent energy would just be massively outweighed by the money investors could make by patenting it and selling it worldwide. (although now that I think about it, from what the episode shows us the game doesn't seem to say that the alga is ready for commercialization or even close, just that people are working on it; some of the shenanigans do make some sense if we assume the technology is still undeveloped, promising enough to warrant suppression by oil companies, but not promising enough that it would attract tons of investors and have backups everywhere; but then there's the question of why future people are even interested in this alga if they never use it in the future, and it was sabotaged before it could get any traction) Anyway, my point is I would have loved to hear you talk more about this stuff. Cynical rants about politics and economy are one of the reasons I love your channel