Coffee Space 
In this article I want to discuss the dangers of having a super intelligence and what that means for the human race, whether it is a bad or good thing and some possible outcomes.
Firstly, lets be clear about what I mean when I discuss super intelligence. By super intelligence, I am speaking about when a machine becomes intelligent enough that coupled with the ability to re-design itself or another, it’s intelligence increases at a completely uncontrolled and seemingly forever more increasing rate. This isn’t just limited to machines, a human could be capable of this too as we learn how the brain works slowly but surely. This discussion doesn’t focus on where this intelligence originates.
Now the real question is: “Is a super intelligence a threat to the human race?”. That’s what everybody wants to know and of course they are right to be thinking about it with seemingly very fast advances in the fields that concentrate on understanding intelligence.
The first stop is “How intelligent could something get?”. If a human is the maximum example of intelligence, the chances are we don’t have to worry. If we’re on a level playing field with anything we consider a threat then generally we’re okay, or at least have a chance. I’m not writing off the ability of something with our intelligence for destructive capability - simply look at our terrible past. Generally, especially in a time where technology is fairly evenly distributed on every side, we are fairly evenly matched with any threat against us.
For this, the first limit we can give where we can start to deduce some hard numbers is a maximum theoretical information density. Somebody who’s name I unfortunately forgot said that if somebody was capable of remembering Graham’s number in their normal sized human head in the decimal numbering system, their head would instantly turn into a black hole. (The reason I say about using a decimal numbering system - if the numbering system is to the base Graham for example then it is trivial to store Graham’s number. As numbers are abstract anyway, you would only have to have a feeling of how big Graham’s number is.) If one really thinks about it, it’s not the information that really causes this but the medium the information is recorded on. If you stored one decimal number per atom, you’d be in trouble. If you stored one decimal per quark I think you would still be in trouble. I think having that much light in any place is pretty bad too.
Then you say “Why not just have a massive brain then, the size of the Universe?”. There is then another issue, how long will it take for two far parts of the brain to communicate? Probably the life time of the Universe, or something measured in the life of the Universe. It may or may not be intelligent, but that’s fairly useless if there isn’t enough time in the Universe to figure yourself out, let alone anything else. As light can now never travel from one side of the Universe to the other due to the expansion of space between the “sides” (whatever that rally means), having a brain the size of the Universe may be completely useless. Perhaps even a brain the size of a galaxy may be difficult with dark matter being extremely mysterious in nature, it could mean large objects even when well structured against collapsing in on themselves could be forever plagued by dark matter.
Then you have another issue on your hands reducing the size down further - you need to have enough stimulation for the parts of the brain not to forget everything. Part of what makes us so intelligent is our ability to forget, allowing us to derive the important information, the important patterns. With this in mind, this great intelligence must have the ability to throw away non-important information and store important information. Some of this knowledge may only be collectable over time. So there must be the ability to forget over periods of time. Human brains forget logarithmically, so it is assumed another intelligence will need to forget in a similar fashion as not to keep useless data.
Input for a 3D brain as we know it is limited to the surface-ish areas. If we model the brain as a sphere like shape for simplicity, we have a surface area of A=4Pir^2 and for volume we have V=(4Pir^3)/3. We want to work out how our input scales with brain size, which would roughly be A/V, giving us:
0001 (4*Pi*r^2)/((4*Pi*r^3)/3) {Our expression}
0002 (12*Pi*r^2)/(4*Pi*r^3) {Times both sides by 3}
0003 (3*Pi*r^2)/(Pi*r^3) {Divide both sides by 4}
0004 (3*r^2)/(r^3) {Divide both sides by Pi}
0005 3/r {Divide both sides by r^2}
And there you have it, as the brain gets larger the amount of possible input decreases. The human brain for example of a brain radius of 1 would receive double the input to volume of a brain of twice the radius. In theory, a brain twice the size of a humans would get twice as board. As the brain radius tends towards infinite size, the amount of input tends towards zero, yet another physical limit to brains.
There may of course be ways around this, there could be a way to directly insert input into the centre by either using a medium that doesn’t effect normal usages (radio waves for example) or using another dimension. Don’t ask me to work that out, just trust me that it allows a larger brain size as the surface area is increased. The larger the brain size, likely the larger number of dimensions required as inputs cross over one another.
Now to the interesting section, imagine this brain doesn’t have an issue with expanding and can get near infinite in size with all the intelligence it needs to do anything… It’s existence wouldn’t mean anything. Part of why we feel as though we belong in the Universe is our ability to interact with it. We have an idea of motion from our ability to move our limbs - and with a little imagination we can stretch that idea to the distances that separate stars. A being with a mind so large, without a body, would struggle to comprehend such a feeling. It would be as real as any other reality it could conjure. Without a body, the ideas remain just ideas, just as valid as any other idea that could be simulated. Without the ability to do experiment, no idea can ever be confirmed. We therefore have another limit, the mind in question must be able to interact with the environment in some meaningful way.