It would be more straightforward to remove the permutations and just display the combinations and the symmetry between heads and tails. And solve it analytically Eg:
if p is the probability that the NPC is correct
P(A|AAAA) = p^4
P(A|BBBB) = (1-p)^4
Anyway, the apparent strangeness of the tie case comes from the fact that the binomial PMF is symmetric with respect to n (the number of participants) and n-k.
PMF = (n choose k) * p^k * (1-p)^(n-k)
So when k = n/2, the symmetry means that the likelihood is identical under p and 1-p, so we're not gaining any information. This is a really good illustration of that; interesting post! (edit: apparently i suck at formatting)
Sailors in the past had a similar adage: “Never go to sea with two chronometers; take one or three.” They relied on precise clocks to calculate longitude.
Bob isn't giving you any actionable information. If Alice and Bob agree, you're more confident than you were before, but you're still going to be trusting Alice. If they disagree you're down to 50% confidence, but you still might as well trust Alice.
In voting parity matters. In some cases, effects are interesting - like in some cases if the mafia/werewolf game, when adding a citizen/villager decreses their winning chance by sqrt(pi/2), vide https://arxiv.org/abs/1009.1031
F T (Alice)
F xx ????????
xx ????????
T ?? vvvvvvvv
?? vvvvvvvv
^ ?? vvvvvvvv
B ?? vvvvvvvv
o ?? vvvvvvvv
b ?? vvvvvvvv
v ?? vvvvvvvv
?? vvvvvvvv
(where "F" describes cases where the specified person tells you a Falsehood, and "T" labels the cases of that person telling you the Truth)
In the check-mark (v) region, you get the right answer regardless; they are both being truthful, and of course you trust them when they agree. Similarly you get the wrong answer regardless in the x region.
In the ? region you are no better than a coin flip, regardless of your strategy. If you unconditionally trust Alice then you win on the right-hand side, and lose on the left-hand side; and whatever Bob says is irrelevant. The situation for unconditionally trusting Bob is symmetrical (of course it is; they both act according to the same rules, on the same information). If you choose any other strategy, you still have a 50-50 chance, since Alice and Bob disagree and there is no reason to choose one over the other.
Since your odds don't change with your strategy in any of those regions of the probability space, they don't change overall.
Let's pretend Alice tells the truth 100% of the time, and Bob is still at 20%. It's more easy to intuit that Bob's contribution is only noise.
Slide Alice's accuracy down to 99% and, again, if you don't trust Alice, you're no better off trusting Bob.
Interestingly, this also happens as a feature of them being independent. If Bob told the truth 20% of the time that Alice told a lie, or if Bob simply copied Alice's response 20% of the time and otherwise told the truth, then the maths are different.
The betting-voting distinction is interesting and was on my mind while I was reading it.
So much of this breaks down when the binary nature of the variables involved becomes continuous or at least nonbinary.
It's an example of a more general interest of mine, how structural characteristics of an inferential scenario affect the value of information that is received.
I could also see this being relevant to diagnostic scenarios hypothetically.
I think there's an annoying thing where by saying "hey, here's this neat problem, what's the answer" I've made you much more likely to actually get the answer!
What I really wanted to do was transfer the experience of writing a simulation for a related problem, observing this result, assuming I had a bug in my code, and then being delighted when I did the math. But unfortunately I don't know how to transfer that experience over the internet :(
(to be clear, I'm totally happy you wrote out the probabilities and got it right! Just expressing something I was thinking about back when I wrote this blog)
I, erroneously, thought that "when Alice and Bob agree there's a 96% chance of them being correct, then surely you can leverage this to get above the 80% chance. What if we trust them both when they agree and trust Alice when they disagree?" Did some (erroneous) napkin math and went to write a simulation.
As I was writing the simulation I realized my error. I finished the simulation anyway, just because, and it has the expected 80% result on both of them.
My error: when we trust "both" we're also trusting Alice, which means that my case was exactly the same as just trusting Alice.
PS as I was writing the simulation I did a small sanity test of 9 rolls: I rolled heads 9 times in a row (so I tried it again with 100 million and it was a ~50-50 split). There goes my chance of winning the lottery!
It depends on what you are doing with the guess. If it is just a question of how frequently you are right or wrong the second person doesn't help. But if you are, for example, betting on your guess the second person improves your odds of coming out ahead significantly, since you can put down a higher wager when they agree than when they disagree.
This kinda reminds me of error correction, and where at some level you can have detectable but not correctable error conditions. Adding Bob is just like adding a parity bit: can give you a good indication someone lied, but won't fix anything. Adding Charlie gives you the crudest ECC form, a repetition code (though for storing one bit, I don't think you can do better?)
I guess if you only need to store one bit you could store either 0 or 11 and on average use less than two bits (for bit flips only), or 111 if you have to also worry about losing/duplicating bits.
One way to think about this is you have a binomial distribution with p=0.8 and n=number of lying friends. Each time you increase n, you shift the probability mass of the distribution "to the right" but if n is even some of that mass has to land on the "tie" condition.
The second liar often gives you information you never wanted! But that is offset by the excellent information of when the liars agree. Fascinating.
I didnt't math during the thinking pause, but my intuition was a second liar makes it worse (more likey to end up 50-50 situation) and additional liars make it better as you get to reduce noise.
Is there a scenario where the extra liar makes it worse, you would be better yelling lalalallala as they tell you the answer?
It would be more straightforward to remove the permutations and just display the combinations and the symmetry between heads and tails. And solve it analytically Eg: if p is the probability that the NPC is correct
Anyway, the apparent strangeness of the tie case comes from the fact that the binomial PMF is symmetric with respect to n (the number of participants) and n-k. PMF = (n choose k) * p^k * (1-p)^(n-k) So when k = n/2, the symmetry means that the likelihood is identical under p and 1-p, so we're not gaining any information. This is a really good illustration of that; interesting post! (edit: apparently i suck at formatting)Sailors in the past had a similar adage: “Never go to sea with two chronometers; take one or three.” They relied on precise clocks to calculate longitude.
Bob isn't giving you any actionable information. If Alice and Bob agree, you're more confident than you were before, but you're still going to be trusting Alice. If they disagree you're down to 50% confidence, but you still might as well trust Alice.
Better than 50% confidence: they only lie 20% of the time, so when they disagree it's still 64% likely to be heads (.8 x .8)
No, it's 50% -- given that e.g. the flip is H, the base probability is both 16% for HT and 16% for TH.
In voting parity matters. In some cases, effects are interesting - like in some cases if the mafia/werewolf game, when adding a citizen/villager decreses their winning chance by sqrt(pi/2), vide https://arxiv.org/abs/1009.1031
When it comes to the wisdom of crowds, see https://egtheory.wordpress.com/2014/01/30/two-heads-are-bett...
I imagined it like:
(where "F" describes cases where the specified person tells you a Falsehood, and "T" labels the cases of that person telling you the Truth)In the check-mark (v) region, you get the right answer regardless; they are both being truthful, and of course you trust them when they agree. Similarly you get the wrong answer regardless in the x region.
In the ? region you are no better than a coin flip, regardless of your strategy. If you unconditionally trust Alice then you win on the right-hand side, and lose on the left-hand side; and whatever Bob says is irrelevant. The situation for unconditionally trusting Bob is symmetrical (of course it is; they both act according to the same rules, on the same information). If you choose any other strategy, you still have a 50-50 chance, since Alice and Bob disagree and there is no reason to choose one over the other.
Since your odds don't change with your strategy in any of those regions of the probability space, they don't change overall.
Let's pretend Alice tells the truth 100% of the time, and Bob is still at 20%. It's more easy to intuit that Bob's contribution is only noise.
Slide Alice's accuracy down to 99% and, again, if you don't trust Alice, you're no better off trusting Bob.
Interestingly, this also happens as a feature of them being independent. If Bob told the truth 20% of the time that Alice told a lie, or if Bob simply copied Alice's response 20% of the time and otherwise told the truth, then the maths are different.
The betting-voting distinction is interesting and was on my mind while I was reading it.
So much of this breaks down when the binary nature of the variables involved becomes continuous or at least nonbinary.
It's an example of a more general interest of mine, how structural characteristics of an inferential scenario affect the value of information that is received.
I could also see this being relevant to diagnostic scenarios hypothetically.
I paused and wrote out all the probabilities and saw no way to improve beyond 80% - I scrolled down hoping to be proven wrong!
(I'm the author)
I think there's an annoying thing where by saying "hey, here's this neat problem, what's the answer" I've made you much more likely to actually get the answer!
What I really wanted to do was transfer the experience of writing a simulation for a related problem, observing this result, assuming I had a bug in my code, and then being delighted when I did the math. But unfortunately I don't know how to transfer that experience over the internet :(
(to be clear, I'm totally happy you wrote out the probabilities and got it right! Just expressing something I was thinking about back when I wrote this blog)
I, erroneously, thought that "when Alice and Bob agree there's a 96% chance of them being correct, then surely you can leverage this to get above the 80% chance. What if we trust them both when they agree and trust Alice when they disagree?" Did some (erroneous) napkin math and went to write a simulation.
As I was writing the simulation I realized my error. I finished the simulation anyway, just because, and it has the expected 80% result on both of them.
My error: when we trust "both" we're also trusting Alice, which means that my case was exactly the same as just trusting Alice.
PS as I was writing the simulation I did a small sanity test of 9 rolls: I rolled heads 9 times in a row (so I tried it again with 100 million and it was a ~50-50 split). There goes my chance of winning the lottery!
It depends on what you are doing with the guess. If it is just a question of how frequently you are right or wrong the second person doesn't help. But if you are, for example, betting on your guess the second person improves your odds of coming out ahead significantly, since you can put down a higher wager when they agree than when they disagree.
The post discusses this exact point near the end.
This kinda reminds me of error correction, and where at some level you can have detectable but not correctable error conditions. Adding Bob is just like adding a parity bit: can give you a good indication someone lied, but won't fix anything. Adding Charlie gives you the crudest ECC form, a repetition code (though for storing one bit, I don't think you can do better?)
I guess if you only need to store one bit you could store either 0 or 11 and on average use less than two bits (for bit flips only), or 111 if you have to also worry about losing/duplicating bits.
This gives me an idea of how to implement isEven() and isOdd() probabilistically.
What happens if Bob lies to Alice 20% of the time and Alice lies to me 20% of the time but I only get input from Alice?
Here i was thinking of transformer architecture
One way to think about this is you have a binomial distribution with p=0.8 and n=number of lying friends. Each time you increase n, you shift the probability mass of the distribution "to the right" but if n is even some of that mass has to land on the "tie" condition.
I wrote a quick colab to help visualize this, adds a little intuition for what's happening: https://colab.research.google.com/drive/1EytLeBfAoOAanVNFnWQ...
Why not unconditionally trust Bob?
You can, but trivially that strategy is also no better than unconditionally trusting Alice.
Better to stay away from lying friends, no?
The second liar often gives you information you never wanted! But that is offset by the excellent information of when the liars agree. Fascinating.
I didnt't math during the thinking pause, but my intuition was a second liar makes it worse (more likey to end up 50-50 situation) and additional liars make it better as you get to reduce noise.
Is there a scenario where the extra liar makes it worse, you would be better yelling lalalallala as they tell you the answer?