I think the really dangerous part here is not just “surveillance bad”.
It is that AI removes the labour cost that used to limit surveillance.
CCTV was already a problem, but someone still had to watch it, search it, interpret it, escalate it. AI changes that. It makes surveillance searchable, scalable and administratively useful. The shift is from “you may be observed” to “your behaviour can be continuously machine-interpreted”.
That changes the moral shape of the state.
A democracy can have police, courts, borders, audits, fraud detection, and public order. I don’t think the serious argument is that no one should ever be watched. The question is asymmetry.
A free society cannot survive if ordinary citizens become more transparent to the state and its contractors than the state is to them.
The principle should be:
privacy for persons, transparency for power.
Police bodycams should make police accountable. Procurement should be inspectable. Algorithmic decisions should have audit trails. Whistleblowers and journalists should be protected. Public systems should be legible to the public.
What worries me is not only some cartoon version of Orwell. It is the boring version: safety dashboards, risk scores, fraud detection, productivity analytics, immigration enforcement, “trust and safety”, compliance automation, procurement contracts.
The boot does not always arrive as a boot. Sometimes it arrives as infrastructure.
And the hard question is not whether surveillance can create order. It obviously can. So can a prison.
The question is whether it creates accountable power afterwards.
A panopticon may produce “best behaviour”, but only by turning citizens into managed subjects. I have been trying to understand this fetish for controlling people through coercion that seems so prevalent in certain new modern business contexts, like amazon warehouse workers and delivery employees.
The only thing it creates is resentment. Is that how you want to build a company or society on. Resentment?
> Is that how you want to build a company or society on. Resentment?
I think this misunderstands their goals. They don't care how society/a company is built. All they care about is that they are the one building it and that they are at the top of the hierarchy.
Just like with the startups and tech companies they built, they see speed as a critical advantage so that they can be the first-mover and establish a moat. Long-term viability and health is a distant secondary or even tertiary concern. If the panopticon and some weirdly neofeudal technofacist society can be built faster than something more egalitarian, then that neofeudal technofacist society is - from their perspective - better and that is what they will bias towards and build.
1. Boring Orwell: Continuous surveillance is already present in the form of cameras in streets, shops, schools, cars, buses, homes, etc. AI can and absolutely will be used to continuously monitor these feeds.
The panopticon concept from Bentham was interesting because even if there was only a small chance that you might be observed, at any given time, then people would act as if they were being observed. Even if they weren't.
We have had that kind of system, now, for just about everything. Not just from the Big Brother direction, but also the Little Brother direction. At any time, a mob of people might decide to pull up your old digital footprint and condemn you for it.
Likewise, even before AI, at any time the IRS could decide to audit your past tax filings, or data breaches could expose your personal secrets, or street camera can nab you for a traffic violation, or someone could decide to pull up surveillance footage and get you for something, and so on.
The exact degree of difference between the two systems is significant, but much of the marginal psychological burden of such things has already been paid by everyone living in industrial civilization. And, as with the panopticon, just the small chances of active monitoring already provided 80% of the sought-after result.
Indeed, that kind of condition is what people like Ted Kaczynski were so bothered by decades ago.
Those living in the epicenters of civilization, like those in the largest cities, have basically been under almost constant surveillance now for decades.
Maybe not "you", but how motivated are you to be in control? Not as much as those who angle for the CEO and board chair roles, or to be kingmakers, or to run for various offices. And they very much tolerate being feared.
It's essentially the TVtropes Fascist but Inefficient, but it takes out the grunt work.[1]
The other thing that comes to mind here is Brazil, the movie directed by Terry Gilliam - the inefficiency of the state is part of what makes it evil because it mostly doesn't care if it gets stuff wrong - I wonder how machine intelligence may change that.
Man, just write your own comments yourself, no need to use AI-generated ones. You are making good point but the twitter-AI-slop style makes it really annoying to read.
Ellison was among several prominent tech billionaires and executives whose names were mentioned in the multi-million-page tranches of documents made public by the Department of Justice.
People at large became surprisingly fine with heavy surveillance state. It’s not even in any election issue agenda, completely ignored by everyone.
Another unpredictable outcome of social media I guess.
Something about stalking people online and digital exhibitionism, pushed the Overton window of surveillance.
To people like me, who do not have such loud and transparent online presence, this is unsettling. I only now crash head first into modern mentality as it is starting to affect me.
I wonder how long will I be able to evade the databases. Up until now it has been not that hard as long as you have a lot of money, live analog life, pay in crypto/cash and avoid big cities.
And of course, the best behavior will be the behavior that poses zero threat to the people in power. What a great future we have ahead of us, my colleagues.
It's subjective. When I was a kid, "best behavior" didn't include assaulting law enforcement and smearing feces on the walls of Congress, but these days that behavior is acceptable, even officially celebrated.
Except China has a well defined culture and millenia of experience with this and has learned how to have an autocratic society that still somewhat serves its people (until it doesn't). The US has been captured by a foreign power that has no interest in the welfare of the people so the expectation is that an autocratic society would be much worse in the US.
And that's what the Larry Ellisons of the world want.
One of the bigger lies ever told is that free-market economies help democratize nations. They don't. If they did, we would have made massive investments in Eastern Europe after the fall of the Marxist-Leninist governments there. Instead we decided to invest in China and, later, Vietnam, among others. These are two very non-democratic nations.
Why did we do that, especially after the Tiananmen Square Massacre? Because it better fit the needs of capital. The last thing they wanted was to set up shop in a bunch of countries where the people had organized against authoritarian regimes for change. If they can get rid of the likes of Ceaușescu and Honecker despite their brutality, they could certainly do things like strike for better working conditions and a greater share of their employers' earnings.
Ellison's just another in a long line of guys with far too much money and far too little empathy. If you get out-of-line in a way that financially or politically inconveniences him, he wants you dealt with as severely as possible. China does that, and now he wants that in the US and other Western countries.
If anyone managed to stick around through the later, lesser seasons of HBO’s Westworld, they were rewarded with a shockingly plausible view of the world Ellison is describing.
And at a time when most of the computing technology required still seemed like sci-fi. I remember kind of chuckling at the idea that the machine intelligence had made and saved a recording of a random conversation Aaron Paul’s character had with his mother in a diner a decade prior.
I have never been a privacy zealot, but it seems inevitable barring major political action that the panopticon will emerge comprehensive, actionable, and cheap.
You mean, you have not created a meet with the relative who could only join remotely and did not transcribe it for the relatives who could not attend? I am shocked /j
It's not the recording that matters, it's the enforcement. In PRC, you get your jaywalking mugshot on the intersection jumbotron, that social humiliation is enough to get people in line. I wager shameless (individualistic) Americans would troll such methods, i.e. it would be counter productive in US cultural contexts... which is bleak because it means would need even harsher big brother enforcement methods to deter. Considering Chinese aunties fisty cuff with cops on the street, in US they'd get ventilated, the kind of shenanigans US LE would have to pull to enforce best behaviors is going to be much more unpleasant.
I think when we look back in 10-20 years, mass nearly universal surveillance will be seen as one of the largest social impacts of AI, or perhaps the largest.
We have barely scratched the surface, and I don't think most people have thought it through.
I guess I appreciate Ellison is educating people about what's going on...
Yeah, LLMs will enable big tech to expand the full surveillance from the online world to the real world, consequently monitoring all aspects of theirs lives.
It's likely gonna take a decade or so for things to become obvious to everyone, just like it took a decade for people to understand the cost of eg. Facebook
People usually don't want to follow traffic laws on road. Now that we have AI camera recording and AI fine system, people try their best to follow traffic laws. We are becoming Japan following traffic laws in certain areas.
Police usually were scared to fine million dollars cars (because the car owner might be part of something higher power and they could lose their job). Now they gossip on roadside and let AI do it's job.
To HN people surveillance is scarry. But to us, surveillance is blessings.
Nvidia has been marketing "smart cities" for years. I've mentioned it several times but it never gets traction. If you think the AI build out is about consumer benefits you're an idiot. It's about control. AI is going to cement the power structures of the world and give authorities powers they've dreamed of since antiquity. It's about surveillance, manipulation and deception.
I work making AI shovels. I benefit a tiny bit from all this but know it's just a small crumb in exchange for the buffet of freedom.
Sounds pretty bad, but what’s the context? We should be skeptical of a quote out of context with some dogpile parallels without any other context. When did he say it? Where did he say it?
One thing I would like clarified is whether he is endorsing that future or just making a neutral prediction on what the state of surveillance will likely be.
Most people are not, indeed. So this is the crux of religion: the "god" watches you, and will punish or reward you. Now social media and governments takes over that role.
If they were just recording and it required human labor to interpret it would merely be a traditional regime of totalitarian repression. But this tech allows real time interpretation. Imagine that a stalwart ideologue gets to decide what behavior constitutes hate, and to dispatch enforcement before you can finish rolling your eyes. This is what makes me fear Ilya Sutskever's warning of an infinitely stable dictatorship. It's not clear that gravity well can be escaped.
It is often commented how people who know they are recorded behave a certain way what is not commented on is how people who know they are recording other people behave. When you see your screen and you know you are recording them it creates this "power trip" it's feeling some of us know and love, but when we truly know it we don't love it anymore it's almost like a drug the high feels good the consequences are for others but that little guilt that lingers we don't like that. Now scale it at the size of an organization, the power trip with other people feel even more better, and the consequences ... well
It worked well in China because the Larry Ellisons of 35 years ago made sure the CCP was rewarded for keeping the rank-and-file in line and working instead of doing things like organizing for better working conditions or agitating for political change.
Sometimes I wonder, if we were actually allowed to engage in the consensual drugs and sex that constitutes 95% of the activity people want to "get away with", would our societal response to infringements on our privacy be even weaker than it currently is?
I think the real risk here is the impact of perfect enforcement of all the tiny things. Enforcement against every minor unsanctioned activity can be self executing with AI. You're going downhill at 10 MPH over the speed limit before you brake, and the camera, if not the vehicle itself, cites you. You cross the street outside of the crosswalk and now you've got 2 strikes. There's nothing to contest, because you acted unlawfully.
What really gets to me is how big tech isn't even pretending anymore to serve society. They clearly feel superior to the rest of us and entitled to rule.
Bryan Cantrill warns, "Do not fall into the trap of anthropomorphizing Larry Ellison. You need to think of Larry Ellison the way you think of a lawnmower. You don't anthropomorphize your lawnmower, the lawnmower just mows the lawn. You stick your hand in there and it'll chop it off, the end. You don't think 'oh, the lawnmower hates me' -- the lawnmower doesn't give a shit about you, the lawnmower can't hate you. Don't anthropomorphize the lawnmower. Don't fall into that trap about Oracle.":
Eh. That quote conflates Ellison and Oracle and I don't think that's correct. I think there's a danger in just accepting that a human being is abhorrent. It _should_ outrage us that Ellison is the way he is. It's silly to think "the lawnmower hates me" because it isn't capable of hate. Ellison is capable of hate and it's not deluded to think he might hate you and I and want to control our lives.
What a strange thing to say. Why would they? Do you have their interest at heart? Do you have the average persons interest at heart? Again: Why would you? That's by definition not how interests work. And that is roughly fine (albeit a kinder universe would be nice).
Even if I have no one's best interest but my own at heart, I can't do anywhere close to as much damage to society at large as a trillion dollar corporation. Also, corporations aren't people, and it's silly to compare individuals to massive organizations.
I'd argue that society ensures (or should ensure) that we, collectively, do have the average person's interest at heart. The problem is when billionaires like these effectively decide to opt out of society. Stop paying taxes, bend the political system to their will.
Even if you think of them as the most selfish people possible, billionaires ought to have the average person's interest at heart. Because once the country is full of average people who are unhappy with their lives it typically doesn't work out well for those with power.
The Golden Rule "do unto others" is an example of this concept, and it answers your question "why would you?"
> And that is roughly fine (albeit a kinder universe would be nice).
You're confusing the status quo with what must be. It's possible for people to have a sane interpretation of self-interest. We should do a better job of reeducating and/or removing from society the people who don't seem capable of that.
There's definitely people who blindly praise and romanticize China and ignore how dystopian all the monitoring systems there are, there is essentially no privacy. Even chat apps are infiltrated or Chinese run in some way, shape, or form.
While that’s true, the Chinese government does also seem to be motivated to keep the cost of housing and food low for its citizens, and provide services and facilities for the public good.
Ha, you really believe that? You can't live in area without the permission of the Government. The current Chinese regime seems to be really going kinda crazy with their new leader. They've lost their more practical leadership since the new guy has consolidated power.
I am not endorsing what China is doing in terms of surveillance by any means, but their argument would be that every nation not subservient to the US got overthrown by CIA backed forces the moment they opened up.
The surveillance tech that is much more likely to be deployed in the US and that few are talking about is Israeli tech used to spy on and suppress the self determination of Palestinians. Especially given the recently proposed fusing of US/Israeli military tech.
Signal genuinely isn't, or other end-to-end encryption chat apps. But yeah, WhatsApp and Discord and others are run by American companies that follow US law.
The real question is whether the threshold before it causes irreversible corruption is before or after the point where you can make real change. The latter is obviously quite terrifying as it essentially means that democracy will always be corrupt (unless time is a factor perhaps)
The way he phrases it makes it clear he doesn't consider himself or his peers as part of the "citizenry". I suppose subjects or peasants or peons might have been a bit too on the nose, even for him.
every time this comes up I feel it's worth reminding people that this already happened before in American history - 1890's and the robber baron industrialists that monopolized everything.
and who can blame them, if we've learned anything from the last few years it's that the American people are utterly servile and passive when faced with these people, in fact they largely seem to crave their wannabe dictators.
In 2023 when Musk refused to sign a collective bargaining agreement with the Swedish IF Metall union, not only did the union resist, not only did other unions in Sweden resist, Scandinavian countries together boycotted supply chains including dock- and metalworkers in Denmark and Norway. What would they do in America? Give him money on Twitter to get a blue checkmark and hope he gives them five minutes of attention in a reply.
If you frame it this way in your mind you will be surprised when people pick surveillance over public disorder. If you don't like that world (I don't either) you can't bury your head in the sand about the problems it is solving, you need a "No, but... " framing where you give answers that actually work.
There is a ground-level appeal of the China-style panopticon because it delivers public order and clean streets. Larry didn't just buy his way to digital dictator by bribing the right people, it answers a question in a way other people are avoiding, because answering it requires a lot of work and uncomfortable tradeoffs.
> There is a ground-level appeal of the China-style panopticon because it delivers public order and clean streets.
Does it? Japan is also famously clean, so maybe it’s the ethnic homogeneity? On the other hand, Singapore is very heterogeneous and clean, and eastern Kentucky is very homogenous and run-down. So maybe this whole attributing outcomes of society to singular factors isn’t very founded in science or reason but just gives people an opportunity to confirm whatever biases they have.
The secret is shame. You can enforce behavior through shame culture slowly (JP) or rapidly with technology, half the reason PRC cleaned up fast is people didn't want their faces plastered on misdemeanour jumbotron. Probably won't work if you live in a shameless society where social humiliation has little or even opposite effect.
Japan has historically been a clean and high-trust society.
China was a ground-level low-trust mess even 25 years ago, with fairly rampant fraud and theft by just about anyone you met on the street. That has completely changed — why? I don't think culture evolves that rapidly ex nihilo, when there's an obvious technology answer.
> There is a ground-level appeal of the China-style panopticon because it delivers public order and clean streets
Clean streets and high speed rail are not a bundled deal with the panopticon - there is no causative linkage. Surveillance - as Ellison plainly mentioned - is for controlling anti-establishment behaviors.
Lookup China subreddits and on YouTube. Tons of Americans wish they could just pick up and move to China. Most Americans that visit China for the first time love it. It is pretty awesome. Have you not been?
Look how well it worked for cops! I'm not suggesting we get rid of their bodycams, but it's easily gamed.
It's one thing to be rich and enjoy the luxuries afforded by that, but to buy dominance over the very lives of others is when the angry mob retaliation is entirely justifiable.
Exactly, cops can impose force on you and their jobs requires them to do it. If they wouldn't have body cams, it would be hard to see if coercion was justified or not. Surveilling powerless folks works in the other direction. It only serves to make them easier to discipline and teaches them that deviations from approved behaviour will be deterministically punished. And I also don't believe anyone is currently paid to act on their best behaviour as a simple citizen. Although this kind of monitoring works quite well for children under 6, you don't want paternalistic institutions, since you will never outgrow them.
From what I've seen it goes both ways with bodycam footage. The cops often have to put up with seriously dangerous stupid people and cops on power trips are video'd abusing people. Not that I am defending Ellison. Sadly the rich will always be able to buy their way out of most any bad situation except screwing over richer people.
You can’t have mass immigration, mainly economic migrants from the third world, unboxed funded police forces, and a legal system built for high trust, highly educated, fairly homogeneous populace at the same time and expect things to be all happy days.
Funny how stuff like privacy and freedom of speech have always been touted as key differentiators of a liberal democracy. And now the mask is falling off because the interests of the ruling class are increasingly diverging from those of the working majority.
He forgot to mention we'll need 1000 more data centers and 3000 new prisons to manage the chaos. So confused by this billionaire behavior with wanting to control society.
It is that AI removes the labour cost that used to limit surveillance.
CCTV was already a problem, but someone still had to watch it, search it, interpret it, escalate it. AI changes that. It makes surveillance searchable, scalable and administratively useful. The shift is from “you may be observed” to “your behaviour can be continuously machine-interpreted”.
That changes the moral shape of the state.
A democracy can have police, courts, borders, audits, fraud detection, and public order. I don’t think the serious argument is that no one should ever be watched. The question is asymmetry.
A free society cannot survive if ordinary citizens become more transparent to the state and its contractors than the state is to them.
The principle should be:
privacy for persons, transparency for power.
Police bodycams should make police accountable. Procurement should be inspectable. Algorithmic decisions should have audit trails. Whistleblowers and journalists should be protected. Public systems should be legible to the public.
What worries me is not only some cartoon version of Orwell. It is the boring version: safety dashboards, risk scores, fraud detection, productivity analytics, immigration enforcement, “trust and safety”, compliance automation, procurement contracts.
The boot does not always arrive as a boot. Sometimes it arrives as infrastructure.
And the hard question is not whether surveillance can create order. It obviously can. So can a prison.
The question is whether it creates accountable power afterwards.
A panopticon may produce “best behaviour”, but only by turning citizens into managed subjects. I have been trying to understand this fetish for controlling people through coercion that seems so prevalent in certain new modern business contexts, like amazon warehouse workers and delivery employees.
The only thing it creates is resentment. Is that how you want to build a company or society on. Resentment?