su3su2u1 posted:

The definition of friendliness they create will just be an abstraction that shares some properties of what we might think of as "friendly."

...

What they really want to create is a sort of "best practices guide to AI development such that it doesn't kill everyone" - that isn't a math problem.

SolTerrasa posted:

I've always thought so; this seems similar to what I've been saying: "it's an engineering problem".

It would be really helpful to understanding what the gently caress they plan to do if they'd release any information about their formalization of the problem.

su3su2u1 posted:

And yea, I was basically saying "I agree it's an engineering problem, which is why formal math is basically useless."

Judging by their publications, they are barely getting to the point where they can formalize "cooperate in the prisoners dilemma" with category theory. An accomplishment they seem abnormally proud of (it's the only publication on the arxiv). It also uses 'timeless decision theory' in that they read the source code of the other bot.

Krotera posted:

Do they get into the halting problem issues there? That seems like the only super interesting part of that particular problem, if they have anything new to say about it.

Krotera posted:

For those playing at home, here's what that means in this context. (the true halting problem is much more general) AI A simulates AI B and does whatever beats B -- B simulates A and does whatever beats A. But they'll compute forever at this rate -- to simulate B, A has to simulate itself and B's reaction -- but to simulate itself for that purpose, it has to simulate itself and B's reaction again, and so on. B has the same problem. Either could stop thinking at a certain recursion depth, but then the other would win by thinking one recursion level deeper. (as it would know by simulating it when the first one would stop) So deciding to terminate is always wrong. Either of these AIs is strong against opponents that can't attempt to simulate *it*, but how do you guarantee that?

Darth Walrus posted:

Expectation had nothing to do with it. Like most of this thread's posters, I stuck my hand in that bear trap voluntarily.

su3su2u1 posted:

And yea, I was basically saying "I agree it's an engineering problem, which is why formal math is basically useless."

Judging by their publications, they are barely getting to the point where they can formalize "cooperate in the prisoners dilemma" with category theory. An accomplishment they seem abnormally proud of (it's the only publication on the arxiv). It also uses 'timeless decision theory' in that they read the source code of the other bot.

The capstone accomplishment of a decade of SIAI is a piece of code that makes it easy to make formal prisoner's dilemma agents play against each other.

Fried Chicken posted:

When you say "their publications" here, do you mean AI researchers taking a bayes approach to it, or Yud and his cult?

quote:

"No, I haven't done Step 2, 'Do an experiment to test your hypothesis.'"

The boy closed his mouth again, and began to smile.

Hermione looked at the drinks can, which she'd automatically put into the cupholder at the window. She took it up and peered inside, and found that it was around one-third full.

"Well," said Hermione, "the experiment I want to do is to pour it on my robes and see what happens, and my prediction is that the stain will disappear. Only if it doesn't work, my robes will be stained, and I don't want that."

"Do it to mine," said the boy, "that way you don't have to worry about your robes getting stained."

"But -" Hermione said. There was something wrong with that thinking but she didn't know how to say it exactly.

"I have spare robes in my trunk," said the boy.

"But there's nowhere for you to change," Hermione objected. Then she thought better of it. "Though I suppose I could leave and close the door -"

"I have somewhere to change in my trunk, too."

Hermione looked at his trunk, which, she was beginning to suspect, was rather more special than her own.

"All right," Hermione said, "since you say so," and she rather gingerly poured a bit of green pop onto a corner of the boy's robes. Then she stared at it, trying to remember how long the original fluid had taken to disappear...

And the green stain vanished!

Hermione let out a sigh of relief, not least because this meant she wasn't dealing with all of the Dark Lord's magical power.

Well, Step 3 was measuring the results, but in this case that was just seeing that the stain had vanished. And she supposed she could probably skip Step 4, about the cardboard poster. "My answer is that the robes are Charmed to keep themselves clean."

"Not quite," said the boy.

Hermione felt a stab of disappointment. She really wished she wouldn't have felt that way, the boy wasn't a teacher, but it was still a test and she'd gotten a question wrong and that always felt like a little punch in the stomach.

(It said almost everything you needed to know about Hermione Granger that she had never let that stop her, or even let it interfere with her love of being tested.)

"The sad thing is," said the boy, "you probably did everything the book told you to do. You made a prediction that would distinguish between the robe being charmed and not charmed, and you tested it, and rejected the null hypothesis that the robe was not charmed. But unless you read the very, very best sort of books, they won't quite teach you how to do science properly. Well enough to really get the right answer, I mean, and not just churn out another publication like Dad always complains about. So let me try to explain - without giving away the answer - what you did wrong this time, and I'll give you another chance."

She was starting to resent the boy's oh-so-superior tone when he was just another eleven-year-old like her, but that was secondary to finding out what she'd done wrong. "All right."

JosephWongKS posted:

Chapter 8: Positive Bias
Part Five


At least Eliezer is self-aware that Eliezarry is condescending and obnoxious. So that’s a start, I guess?

No wait a minute, there’s also the bit about “that was secondary to finding out what she’d done wrong”. That implies that Eliezer thinks that if someone is being condescending to you, the onus is on you to be condescended to in order to find out “what you’d done wrong”. He’s merely justifying Eliezarry’s being so unlikable and annoying.

Pvt.Scott posted:

I'm not sure I see the problem in the catgirl scenario. Like, if you get tired of your furry fuckdolls, just leave the volcano lair and hang out with some of the other people in paradise.

quote:

He said, "Well, then I'd just modify my brain not to get bored—"

SSNeoman posted:

Right my mistake. It's actually 3^^7625597484987 which is some godawful number you get if you take 3 and then raise it to the 3rd power 7,625,597,484,987 times which is still the same thing as "a meaninglessly big number" which means my point still stands.

SSNeoman posted:

But you know what fine, whatever. Lemme explain why Yud's conclusion is nuts.
So there is a branch of philosophy called utilitarianism. The basic premise is that it wants to achieve the maximum amount of happiness for the maximum amount of people (in broad strokes).
In Yud's PHILO101 he blasts the problem with a blunt solution. Dude is tortured for 50 years, or someone is tortured (ie: removes the dust speck) for like 1 second? Well obviously we'd pick option 2. But when you SHUT UP AND MULTIPLY our second by that "meaninglessly big number", suddenly having one dude tortured for 50 years doesn't sound that bad, right? BEEP BOOP PROBLEM SOLVED GET hosed CENTURIES OF PHILOSOPHY

Well of course not. There are other factors involved when you realize that we are dealing with a being who has life, willpower, the ability to feel pain and all that other good poo poo. We are taking 50 years out of a person's life and replacing it with pain and misery, instead of inconveniencing a lot of people for an insignificant amount of time. This logic, by the way, is still equating torture with the pain caused by dust specks. I am still playing by Yud's rules despite the fact the two are obviously not equal. You are crushing a person's dreams and ambitions just for the sake of not bothering a whole lot of people for something they won't remember. What about the life this poor person is missing out by going through this torture? All those people won't even remember the speck by the end of the day, but that person will carry his PTSD for the rest of his (no doubt shortened) life, if he's not loving catatonic by the end of the first year.
I am explaining this in-depth because it's a problem that requires you to do so. It is something LW themselves avoid doing, and I hope you are not falling into the same trap.

su3su2u1 posted:

I mock Yudkowsky from a position of strength rather than of ignorance- I have a phd in physics, and Yud included (wrong) physics references in his fanfic. He incorrectly references psychology experiments in his fanfic. He uses the incorrect names for biases in his fanfic. He gets computational complexity stuff wrong in his fanfic. He does this despite insisting on page 1 that "All science mentioned is real science." Let me ask you- if an AI researcher can't get computational complexity correct, why should I trust anything else he writes? If someone who has founded a community around avoiding mental biases can't get the references right in a fanfic, why should I trust his other writing?

His technical work is no better. His paper "timeless decision theory" paper is 100 pages of rambling, with no formal definition of the theory anywhere (and it would be super easy to formalize the theory as described). His research institute is a joke- they've been operating for more than a decade with only 1 paper on arxiv and basically no citations to any of their self-published garbage.

akulanization posted:

akulanization posted:

I have no evidence this conversation actually occurred, but if it did that still doesn't change my point. Harriezer is held up as an example, he's powerful and has agency because he is a rationalist. He is obviously meant to inspire people to be rationalists like himself, a perception that isn't helped by the author saying poo poo like this:

akulanization posted:

Also the actual number is immaterial if you want to approach this problem from the perspective of mathematics, it's an arbitrarily large real number. If you grant the premise that both events are measured with the same scale and that minimizing that number is a priority, then the number doesn't matter; you can always select a number big enough that the "math" works out the way you want. The problem with the torture proponent's stance is that, as SSNEOMAN pointed out, they don't actually consider the problem. They wipe out a life because that's better than a trifling inconvenience to a huge number of people.

petrol blue posted:

Yeah, the 'position of ignorance' line probably wasn't a good move, Legacyspy, goons like physics and ai almost as much as Mt. Dew.

Nessus posted:

Here you go, buddy! http://forums.somethingawful.com/showthread.php?threadid=3627012

The idea is that the AI is going to inevitably become God, so it is the most important thing ever to make sure that when the computer inevitably becomes Literally God, we make sure it's a nice friendly New Testament God who cares for us, rather than an Old Testament God who will send us all to Robot Hell.

Night10194 posted:

I mock the writing in the story because it's bad and we're on somethingawful.com,

Night10194 posted:

but Yud's cult is legitimately fascinating to me as a religious studies guy who is interested in getting into studying emerging religions, and so his proselytization-fiction is actually really interesting from an academic standpoint.

Legacyspy posted:

This is what I mean. If Nessus is just being sarcastic or w.e that is fine. But if he honestly thinks that friendly A.I is a worry over A.I torturing us... then he doesn't understand what, right or wrong, Eliezer is talking about. Its a worry that the A.I, in pursuit of the goals we gave it, may have unintended consequences that could be bad for us. This can be as simple as being a lovely A.I that when asked "How do we get rid of insects eating our sugar cane crop" says "introduce the cane toad" not understanding that the consequences of cane toad infestation will be far more annoying than the insects eating our crops. Or an A.I that does for some reason we can't couldn't have foreseen decide to "kill all humans" in pursuit of its goals. I think this is unlikely but I don't think the idea of "How do we get an A.I to recommend courses of actions that take into account the complex values we have (like not liking a cane toad infestation) is a useful one. Whether Eliezer is actually doing anything useful on this front, I can't tell. But afaik hes also one of the few people even talking about it.

Night10194 posted:

I'd define it primarily as people who are actual contributors to his institute (financial ones), and who are members of the Less Wrong community. I have no idea if you're a member. You're pretty obviously a fan of the work, but that has no real bearing on if you self identify as a member of the community or of Yudkowsky or similar 'rationalist' orbits. My primary academic interest is in the fact that this fiction, and the Sequences, and many of his theories, have a cast very similar to a lot of Christian religious and apocalyptic dogma, despite their avowed atheism. I'm currently beginning to gather data and do reading on his work because of the fascinating parallels between the Cryonics stuff and the Christian resurrection of the Dead, the similarities between AI Go Foom and classic apocalypse, etc, because I have approval and support from my old advisers from my master's program that there might be a productive bit of work to be done on singularity and science fetish cults, and on the sort of cross pollination between commonplace religious ideas in the larger culture and the texture of what they end up believing.

I'm at the very beginning of working on this, mind, and have a hell of a lot of reading to do still. Just some of the ideas and the general shape of things piqued my interest in their similarity and merit looking into from an academic standpoint.

Legacyspy posted:

Sure. You can absolutely argue that dust-specks is the preferable choice. The point of the question though is of consistency. If one says that there is no sufficiently large number where torture is preferable to dust specks, then to be logically consistent you must make sacrifices elsewhere. But most people don't behave in a way that is logically consistent with a preference to dust over specks. For example, they drive cars distances they could walk, a 1 / (some number) chance of causing other people serve harm, to save them trouble of walking to the grocery store ( a minor inconvenience). Of course most people agree that "some number" is so big that the chance of causing enough harm to worry about is not worth the trouble of driving to the grocery store.

quote:

I am in no position to argue whether or not anything he has done w.r.t to A.I is legitimate. And I didn't mean you guys were ignorant of science or anything. I just meant a lot of people were mocking stuff that hadn't even personally read.

quote:

Yud's cult? Lesswrong is more or less dead. The only active following Yudoswky has right now is /r/hpmor and that will slowly die off since nothing is being written. That is not say there isn't a culture of people who share similar ideas to Yud, the "rationalist" culture or w.e. But I don't think they "belong" to Yud. I'm curious. How would you identify someone in his cult? Am I in his cult?

Tiggum posted:

Saying that no one else is even talking about this problem is like saying no one's talking about the problem of interplanetary diplomacy. Since there are no other inhabited planets to communicate with right now, it's not exactly a pressing question. If we end up colonising other planets or meeting another intelligent species then it'll be relevant, but until then we don't even know the parameters we'd be working within, so it's fairly useless to come up with any "solutions" just yet.

Legacyspy posted:

But this sounds like a totally interesting thing to explore.

Tiggum posted:

Sure, it's a great premise for science fiction. And there's tons of sci-fi about interplanetary diplomacy (and AGIs). There just aren't many people talking about it as a practical issue, because it isn't one.

quote:

The boy's expression grew more intense. "This is a game based on a famous experiment called the 2-4-6 task, and this is how it works. I have a rule - known to me, but not to you - which fits some triplets of three numbers, but not others. 2-4-6 is one example of a triplet which fits the rule. In fact... let me write down the rule, just so you know it's a fixed rule, and fold it up and give it to you. Please don't look, since I infer from earlier that you can read upside-down."

The boy said "paper" and "mechanical pencil" to his pouch, and she shut her eyes tightly while he wrote.
"There," said the boy, and he was holding a tightly folded piece of paper. "Put this in your pocket," and she did.

"Now the way this game works," said the boy, "is that you give me a triplet of three numbers, and I'll tell you 'Yes' if the three numbers are an instance of the rule, and 'No' if they're not. I am Nature, the rule is one of my laws, and you are investigating me. You already know that 2-4-6 gets a 'Yes'. When you've performed all the further experimental tests you want - asked me as many triplets as you feel necessary - you stop and guess the rule, and then you can unfold the sheet of paper and see how you did. Do you understand the game?"

"Of course I do," said Hermione.

"Go."

"4-6-8" said Hermione.

"Yes," said the boy.

"10-12-14", said Hermione.

"Yes," said the boy.

Hermione tried to cast her mind a little further afield, since it seemed like she'd already done all the testing she needed, and yet it couldn't be that easy, could it?

"1-3-5."

"Yes."

"Minus 3, minus 1, plus 1."

"Yes."

Hermione couldn't think of anything else to do. "The rule is that the numbers have to increase by two each time."

"Now suppose I tell you," said the boy, "that this test is harder than it looks, and that only 20% of grownups get it right."

Hermione frowned. What had she missed? Then, suddenly, she thought of a test she still needed to do.

"2-5-8!" she said triumphantly.

"Yes."

"10-20-30!"

"Yes."

"The real answer is that the numbers have to go up by the same amount each time. It doesn't have to be 2."

"Very well," said the boy, "take the paper out and see how you did."

Hermione took the paper out of her pocket and unfolded it.

Three real numbers in increasing order, lowest to highest.

Hermione's jaw dropped. She had the distinct feeling of something terribly unfair having been done to her, that the boy was a dirty rotten cheating liar, but when she cast her mind back she couldn't think of any wrong responses that he'd given.

"What you've just discovered is called 'positive bias'," said the boy. "You had a rule in your mind, and you kept on thinking of triplets that should make the rule say 'Yes'. But you didn't try to test any triplets that should make the rule say 'No'. In fact you didn't get a single 'No', so 'any three numbers' could have just as easily been the rule. It's sort of like how people imagine experiments that could confirm their hypotheses instead of trying to imagine experiments that could falsify them - that's not quite exactly the same mistake but it's close. You have to learn to look on the negative side of things, stare into the darkness. When this experiment is performed, only 20% of grownups get the answer right. And many of the others invent fantastically complicated hypotheses and put great confidence in their wrong answers since they've done so many experiments and everything came out like they expected."

Legacyspy posted:

There is nothing ironic about it. I said Artemis was a character like Harry in reply to a comment by Nessus first saying that Artemis was a character like Harry, and to my limited knowledge that seemed correct. If you want to argue whether or not Artemis is like Harry, argue with Nessus, not me. I don't know the books well enough.

Legacyspy posted:

Why not just admit you were wrong instead of backtracking?

Legacyspy posted:

Well I'm too lazy to screen shot the reddit P.M because you'd probably just say that wasn't sufficient evidence (Photoshop and all). You can just P.M him yourself on Reddit if you want. And it doesn't change your point? Why not just admit you were wrong instead of backtracking? I definitely don't disagree that he was meant to inspire people or lead them to the sequences. He was clearly intended to. However he was clearly not intended to be some sort of super rationalist, as you and others argued.

Legacyspy posted:

I agree the actual number is immaterial. I was just using the actual number to demonstrate that the he clearly hadn't read the original thing Eliezer had written. And the problem with the "dust speck people" is that if they are consistent in their preferences, then they must make sacrifices elsewhere. Only most people are not consistent and would not make these sacrifices if asked.

Legacyspy posted:

This is what I mean. If Nessus is just being sarcastic or w.e that is fine. But if he honestly thinks that friendly A.I is a worry over A.I torturing us... then he doesn't understand what, right or wrong, Eliezer is talking about. Its a worry that the A.I, in pursuit of the goals we gave it, may have unintended consequences that could be bad for us. This can be as simple as being a lovely A.I that when asked "How do we get rid of insects eating our sugar cane crop" says "introduce the cane toad" not understanding that the consequences of cane toad infestation will be far more annoying than the insects eating our crops. Or an A.I that does for some reason we can't couldn't have foreseen decide to "kill all humans" in pursuit of its goals. I think this is unlikely but I don't think the idea of "How do we get an A.I to recommend courses of actions that take into account the complex values we have (like not liking a cane toad infestation) is a useful one. Whether Eliezer is actually doing anything useful on this front, I can't tell. But afaik hes also one of the few people even talking about it.

su3su2u1 posted:

You could go one step further and say everyone could choose not to drive to save some lives, but choosing not to drive would also cost lives.

su3su2u1 posted:

There is no inconsistency in choosing driving to the store over walking but also choosing dust specks over torture.

su3su2u1 posted:

Your comparison is bad- choosing torture over dust specks is giving one person a certainty of ruining a life vs lots of minor inconveniences.

akulanization posted:

I am desperately trying to remain civil in this discussion, you are making it very hard. You posted in support of a position, when you were presented with an argument or evidence that the position was incorrect you claimed to have never held the position in the first place.

quote:

So, first of all it was raised by Nessus, not me, that said Harry is like Artemis. I could remember some similarity so I rolled with.

akulanization posted:

You don't appear to grasp that Harriezer may not be the infini-rational from the perspective of Big Yud, but in the story he is the only "rationalist" and he uses that to go on, and on, and on about how dumb people are and how they need to think more like him. He is definitely the High King of Rational Mountain because there are no challengers to the throne. When Yud goes, "well he isn't as rational as I am " He comes off as a hack who is trying to post hoc rationalize the mammoth failings of his character. I mean he has had years to come up with excuses for his terrible pacing and characterization.

akulanization posted:

You are doing the thing where you contradict yourself, again. If the number is immaterial to the argument or the counter argument then obviously it does not matter if it was quoted correctly. Since both the argument for torture and the argument against it do not rely on a precise value of N, then there is no use distinguishing between ~10^38 and an even bigger number. So if you are trying to argue that they didn't read Yud, it's trivial to show they did since they loving quoted him. And if you are arguing they didn't understand him, then you need to actually show how a component they missed addresses or renders moot some part of their argument; the precise value of a stupidly huge number isn't doing that.

"Legacyspy" posted:

examples of consequences of preferring dust specks to torture.

petrol blue posted:

I sat down and thought about the best choice for ages, invalidating the benefit of choosing optimally.

Legacyspy posted:

The point is, all you need to do to get a world where someone is tortured (very bad consequence) to avoid a minor inconvenience across a sufficiently large number of people, is have a sufficiently large number of people such that the probability of at least one person getting a very bad consequence is near unity. And that is why people who prefer dust speckers are inconsistent.

Legacyspy posted:

The point is, all you need to do to get a world where someone is tortured (very bad consequence) to avoid a minor inconvenience across a sufficiently large number of people, is have a sufficiently large number of people such that the probability of at least one person getting a very bad consequence is near unity. And that is why people who prefer dust speckers are inconsistent.

Added Space posted:

I think he's saying that, given enough dust specks, people are going to lose eyes or get distracted and die. If you alter the problem in such a way, the torture does start to look more preferable.

platedlizard posted:

Nah, because torture is a deliberate act of evil and accidents/eye injuries from dust specks aren't. Accidentally hitting a child on the road because something interfered with your vision sucks for you and the child, but it's isn't the same level of awful as taking that kid and torturing them for 50 years. A person has to be a real rear end in a top hat to think they are at all comparable imo

Nessus posted:

What I also notice is that this 3^^^^3 figure or whatever is so ridiculously huge that it becomes absurd, and I think it's because Yud realizes that if you did the numbers for any semi-sane number, such as "every human who's ever lived" (which would probably be a hundred billion tops?) or "all the inhabitants of a thousand Earths" (which we could probably top out at ten trillion), the effects would be meaningless.

What I also don't understand is what the hell this is supposed to prove, exactly. Like what's the theological point of the dust-speck thing? The Nessus seeks to understand this foolishness.

Nessus posted:

What I also notice is that this 3^^^^3 figure or whatever is so ridiculously huge that it becomes absurd, and I think it's because Yud realizes that if you did the numbers for any semi-sane number, such as "every human who's ever lived" (which would probably be a hundred billion tops?) or "all the inhabitants of a thousand Earths" (which we could probably top out at ten trillion), the effects would be meaningless.

What I also don't understand is what the hell this is supposed to prove, exactly. Like what's the theological point of the dust-speck thing? The Nessus seeks to understand this foolishness.

Legacyspy posted:

Is there a number large enough such that you wouldn't pay a dollar to prevent a 1/(large number) chance of preventing something terrible from happening? Or do you really live your life taking any minor inconvenience that prevents no matter how negligible, a very, very small chance of something bad happening to you?

Everyone doesn't prefer a minor inconvenience which would prevent a 1/(big enough number) chance of something happening them, for just taking that chance instead. Do you eat bananas? Would not eating bananas (or other foods similarly radioactive) consist of a minor inconvenience (if not just substitute something else you consider a minor inconvenience and pretend it too has a similarly low chance of a really bad consequence)? But bananas contain potassium that could decay and in the process emit a particle that strikes the DNA of one your cells, making it cancerous, which by chance doesn't get caught in time by your body, and now you have cancer.

The point is, all you need to do to get a world where someone is tortured (very bad consequence) to avoid a minor inconvenience across a sufficiently large number of people, is have a sufficiently large number of people such that the probability of at least one person getting a very bad consequence is near unity. And that is why people who prefer dust speckers are inconsistent.