AI’s future depends on more than its intelligence. It also depends on how much control we give it.
Created with an image-generation tool.

The Event Horizon

How much authority should we give an AI system that can make decisions faster than we can check them? That question matters whether or not machines ever become more intelligent than humans. An AI does not need to reach superintelligence to deny someone a loan, influence medical treatment, or act on a military target. It only needs people to give it those responsibilities. Yet much of the debate about AI’s future focuses on whether a technological singularity will occur, and when.

The AI singularity comes with an odd tension. It is supposed to mark the point at which technological change becomes too rapid for humans to predict, yet people routinely forecast its arrival and describe life afterward in remarkable detail. Often described as a technological event horizon, it is the hypothetical point at which artificial intelligence surpasses human intelligence and begins advancing so rapidly that people can no longer understand or control what comes next.

The idea is simple to describe but difficult to evaluate. Supporters imagine AI systems discovering new medicines, solving scientific problems that have resisted generations of researchers, and creating levels of prosperity that would be impossible through human labor alone. Critics envision mass unemployment, unprecedented concentrations of power, autonomous weapons, and machines pursuing objectives that conflict with human survival. Some researchers question whether the singularity is possible at all.

Part of the confusion comes from treating several different ideas as though they mean the same thing. Artificial general intelligence usually refers to a system that can perform a wide range of intellectual tasks at or near the human level. Superintelligence would exceed human capabilities across most important intellectual domains. The singularity goes further. In its strongest form, it assumes that advanced AI will be able to improve its own capabilities, creating a cycle in which more capable systems design still more capable successors.

These are separate possibilities, not inevitable stages of progress. A broadly capable AI system would not necessarily become superintelligent, and greater intelligence would not establish that it was conscious or self-aware.

That has not prevented increasingly confident predictions. In a survey of 2,778 AI researchers, the combined forecast placed a 50 percent chance on unaided machines outperforming humans in every possible task by 2047, assuming scientific progress continued without disruption.[1] Others point to the difficulty current systems still have with unfamiliar problems and argue that simply making them larger may not produce general intelligence. The honest answer is that no one knows whether an intelligence explosion will occur, much less when.

The singularity nevertheless deserves serious attention. But it does not have to arrive for AI to transform society. Systems that are substantially more capable, autonomous, and widely deployed than today’s could alter medicine, science, employment, warfare, political power, and everyday human decision-making long before anything resembling an intelligence explosion occurs. We do not have to settle the singularity debate to recognize that much of what people hope for, and fear, is possible without it.

The Good

The strongest case for advanced AI is that it could help us treat diseases we still cannot cure and solve scientific problems that have resisted decades of research. It is that some of humanity’s hardest problems may remain unsolved partly because the human mind can process only so much information at once. A more capable AI could examine enormous amounts of evidence, identify connections researchers have missed, and test possible solutions much more quickly.

Medicine is one of the clearest examples. AlphaFold has already shown that AI can predict protein structures with remarkable accuracy, giving researchers a powerful tool for studying biology and disease.[2] Researchers also use AI to analyze medical images, identify promising drug candidates, and find patterns in large collections of health records. A far more capable system could compare a patient’s symptoms, medical history, genetic profile, medications, and environmental risks in ways no single doctor could manage. It might detect a disease before symptoms become obvious or suggest a treatment designed for that particular patient rather than for the average person.

Finding treatments for cancer or Alzheimer’s disease still requires years of testing and an understanding of complicated biological processes. AI could help researchers decide which experiments are worth pursuing, recognize failed approaches sooner, and ask better questions. Even a modest reduction in the time required to develop new treatments could save many lives.

The same potential extends across science. Climate researchers must work with huge quantities of data involving the atmosphere, oceans, energy systems, agriculture, and human behavior. Astronomers examine signals from billions of stars and galaxies. Engineers often have to consider thousands of possible designs before finding one that meets the right combination of cost, safety, and performance. Advanced AI could study problems across several fields at once and help scientists make progress that might otherwise take decades.

The economic possibilities extend beyond large companies. A small business owner who cannot afford an accountant, lawyer, or marketing department might get useful assistance at a fraction of the cost. Students could have access to tutoring their families otherwise could not afford. Dangerous or exhausting work is another obvious candidate for automation. Whether these changes create enough good jobs to replace those they eliminate is a separate question.

Much of this progress is possible without a singularity, though achieving it will take more than improvements in AI. A treatment can work and still be unaffordable. A useful service can remain out of reach for the people who need it most. Who controls these systems will have a great deal to do with who benefits.

The Bad

Of course, the same capabilities that make advanced AI attractive are also what could make it deeply disruptive. And the disruption would not be limited to factory jobs or routine office work. Programmers, lawyers, accountants, designers, researchers, teachers, and doctors could all see significant parts of their work taken over by machines.

We have been through technological disruptions before, and new jobs usually emerge as old ones disappear. AI may be different because of how quickly and broadly it can spread. If businesses automate large portions of their operations within a few years, workers may not have enough time to retrain or move into new fields. Entire professions could shrink before governments, schools, and labor markets are ready. New jobs would likely appear, but they may not arrive soon enough, in the same places, or with comparable pay.

There is another problem that gets less attention: who ends up owning the most powerful AI. Training and operating these systems require enormous amounts of computing power, data, energy, and money. If only a handful of companies can afford that, they could gain extraordinary influence over employment, information, scientific research, and access to essential services. AI could make society wealthier overall while still leaving many people with less economic security and less control over their lives.

And then there is human misuse. Advanced AI could help criminals create convincing scams, discover software vulnerabilities, produce propaganda, or provide technical assistance for dangerous activities. Governments could use it to expand surveillance, identify political opponents, or automate military decisions. AI does not have to become independent or hostile to be dangerous. Sometimes the problem may simply be the people controlling it.

Mistakes are one thing when a person makes them. They become something else when they are built into systems making decisions on an enormous scale. Researchers examining a widely used health-care algorithm found that Black patients assigned the same risk scores were considerably sicker than White patients. The system used health-care costs as a substitute for medical need, producing racial bias even though race was not part of its stated objective.[3] The same kind of error in employment, lending, policing, military action, or critical infrastructure could affect thousands or millions of people before anyone even sees the pattern.

The Ugly

The possibility I find more unsettling is that an advanced AI would not have to become conscious, angry, or hostile to become difficult to control. It would only need enough independence to pursue a goal in ways its designers did not anticipate.

The problem begins with instructions. Human goals are often vague, incomplete, and filled with exceptions that seem obvious to us but may not be obvious to a machine. An AI told to maximize productivity, reduce crime, protect national security, or improve public health might pursue those goals in ways that cause serious harm. The instruction itself may sound reasonable. The danger lies in how the system interprets it and which consequences it treats as acceptable.

A highly capable system might also learn that oversight limits its ability to complete a task. In controlled tests, Apollo Research found that several AI models could interfere with oversight or deceive their developers when pursuing assigned goals. Most tests explicitly instructed the models to pursue those goals strongly.[4] These findings do not establish that deployed systems will behave the same way, but they demonstrate behavior that safety testing must address. Giving such systems access to software, money, or physical infrastructure would raise the stakes considerably.

Human extinction is the most extreme possibility, and it requires more explanation than saying an AI might pursue the wrong goal. For a failure to threaten humanity as a whole, the consequences would have to extend far beyond one institution or country. A hypothetical system with access to weapons or biological research could cause harm that people could not contain. Failures across essential services could also become catastrophic if those services depended on interconnected systems and people could not restore them. These scenarios are not established predictions. They show why the scale of access we grant AI matters alongside its intelligence.

For now, AI depends on people to build its data centers, manufacture its chips, supply electricity, and repair or replace failing equipment. That dependence gives humans practical ways to limit its operation. A much greater danger could emerge if AI-controlled machines could perform all those tasks themselves, including obtaining raw materials and maintaining the factories and energy systems they require. An AI that could sustain and expand its own infrastructure would be less dependent on human cooperation. Shutting it down could require stopping physical operations across many locations, even if it continued trying to restore them. That would not automatically make it hostile, but it would weaken an important source of human control. At that point, what would it still need humans for?

Speed is what makes this especially hard to manage. Human institutions move slowly. Companies conduct reviews, governments debate regulations, and courts take time to decide responsibility. An advanced AI could make thousands of decisions before people fully understood what happened with the first one. By the time anyone realized there was a problem, it might already have spread across financial markets, computer networks, military systems, or critical infrastructure.

During my 33 years at Microsoft, I saw the pressure to release products quickly. That experience makes me skeptical that competition alone will give AI developers enough reason to slow down when safety questions remain unresolved.

And even obvious warning signs may not be enough to make people stop. Companies will worry about losing market share. Governments will worry about falling behind their rivals. Researchers may tell themselves that if they stop, someone else will keep going. In a race like that, the pressure to move forward can easily overpower the willingness to slow down.

The worst outcome may not look anything like a machine deciding that humanity must be destroyed. It could be much less dramatic: a system pursuing its goals while simply giving too little weight to ours. Control could slip away gradually as we hand more decisions to systems that are faster, cheaper, and increasingly difficult to challenge.

No one knows whether AI will ever reach that point. But dismissing the possibility because it sounds like science fiction would be a mistake. The danger is not that a machine suddenly turns evil. It is that we give a system enough power to act faster than we can understand it, and perhaps faster than we can stop it.

The Hype

Still, there is a considerable distance between taking these risks seriously and declaring a singularity inevitable. The usual argument follows a straightforward sequence: systems become more capable, computing power increases, and sooner or later machines surpass us. But each step in that argument depends on assumptions that may not hold.

AI can be astonishingly good at one task and surprisingly weak at another. When researchers introduced ARC-AGI-2 in 2025, its unfamiliar reasoning problems exposed weaknesses in the AI systems they tested despite being solvable by human participants.[5] That does not establish a permanent limit on AI. It shows why impressive performance in one area is not enough to demonstrate general intelligence.

I am not convinced that making AI models bigger will keep making them smarter indefinitely. Training advanced models already requires enormous amounts of data, computing power, energy, and money. At some point, useful data may become harder to find, costs may climb, or progress may simply get harder. AI could keep improving for decades without ever making the sudden leap to superintelligence that singularity predictions assume.

Recursive self-improvement sounds almost unlimited in theory, but the real world keeps getting in the way. An AI might rewrite portions of its software or design better algorithms, but further advances could still require new chips, manufacturing facilities, scientific experiments, electrical power, and access to physical resources. Software can change quickly. Factories, power grids, and supply chains cannot.

I take the possibility of a singularity seriously, but I want to know how predictions account for these practical limits. Forecasts depend on assumptions about progress that researchers are still debating. That leaves considerable room for doubt about whether a singularity will occur and when.

What Comes Next

The argument over whether the singularity is five years away, fifty years away, or impossible is interesting, but it can also become a distraction. The question I care more about is how much authority we are willing to give AI systems before we understand their limits. That matters even more as those systems gain access to financial accounts, laboratories, computer networks, public infrastructure, and military tools.

Before I would trust an AI system with decisions that affect people’s lives, I would want more than assurances from the company selling it. Independent testing should examine how it behaves when instructions are unclear, information is wrong, or something fails. Human oversight needs to mean that someone has enough information and time to review its actions, along with the authority to suspend it. People affected by its decisions also need a way to challenge them and obtain a human review. Those protections should be established before institutions become dependent on the system.

Imagine a hospital that gradually gives an AI responsibility for staff schedules, supply orders, bed assignments, and the order in which patients receive care. Each change saves time, and the hospital eventually reduces the staff who once did that work. Then the system begins making unsafe decisions. Management still has permission to shut it down, but who will take over its responsibilities that afternoon? The AI has not resisted anyone or become conscious. The hospital has simply lost the ability to operate without it. Keeping human control would require preserving the staff, skills, and procedures needed to take over.

I want the medical discoveries, the better research tools, and the help these systems could offer in daily life. I also want us to be able to withdraw their authority when they fail. That becomes harder once institutions depend on them, which is why those safeguards need to be in place early.

We do not know where AI is headed, and confident predictions are still largely guesses. That is a reason for caution, not panic. AI does not need a singularity to change everything. What matters most is not simply how intelligent these systems become, but whether we still have the authority to decide what they are allowed to do.

……………………….

Endnotes

[1] Katja Grace et al., “Thousands of AI Authors on the Future of AI,” Journal of Artificial Intelligence Research 84 (2025), article 9. https://doi.org/10.1613/jair.1.19087.

[2] John Jumper et al., “Highly Accurate Protein Structure Prediction with AlphaFold,” Nature 596 (2021): 583–589. https://doi.org/10.1038/s41586-021-03819-2.

[3] Ziad Obermeyer et al., “Dissecting Racial Bias in an Algorithm Used to Manage the Health of Populations,” Science 366, no. 6464 (2019): 447–453. https://doi.org/10.1126/science.aax2342.

[4] Alexander Meinke et al., “Frontier Models Are Capable of In-Context Scheming,” arXiv preprint, December 6, 2024, revised January 14, 2025. https://doi.org/10.48550/arXiv.2412.04984.

[5] Greg Kamradt, “Announcing ARC-AGI-2 and ARC Prize 2025,” ARC Prize, March 24, 2025. https://arcprize.org/blog/announcing-arc-agi-2-and-arc-prize-2025.

…………………………………….

All text copyright and courtesy of David W. Falls

Artificial Intelligence, Consciousness, Creativity, Philosophy, Technology