global-e header

Demonstrating the contact tracing app "TraceTogether" in Singapore in March 20, 2020.  (Photo credit: The Straights Times)

Demonstrating the contact tracing app "TraceTogether" in Singapore in March 20, 2020.  (Photo credit: The Straights Times)

The Promises and Pitfalls of Algorithmic Governmentality

October 6, 2020|Volume13 |Issue67

The COVID-19 pandemic has been foremost a global health crisis but it has also clearly become a political crisis for many democracies. People around the world have hoped for political, economic, environmental, and social change to grow out of the epidemic and its many side effects. The reality has turned out to be different, and disappointing. 
 
On the socio-political level, analysts observe the return of the nation-state ideal: closed borders, subsidized economies, and increased control over individuals and their bodily movements. However, it seems that the coronavirus pandemic also suggests a gradual disappearance of democratic governance as we have known it in favor of a rather new type of governance: algorithmic governmentality (Berns & Rouvroy). The debates on Artificial Intelligence as related to infection-tracing applications are representative of this transition in forms of exercising power. This allows us to consider a re-globalization of socio-political governmentality, which may take place after the global health and political crises have passed and which may become their most important outcome in the long term. From a governmentality implemented on national populations through compiling statistics to a governmentality implemented on citizens-users through big data and algorithms, we are witnessing a global reframing of the exercise of power based on a GAFAM (Google-Amazon-Facebook-Apple-Microsoft) model. 
 
Reframing Governmentality through AI Technologies
Governmentality is defined by Michel Foucault as the expression and form of the capacity of states in exercising power to govern populations through rationality. If states throughout history have always had certain time-adequate technological and governance devices (Supiot, 2012) to implement their policies (such as laws, statistical tools, educational standards), it is now the technologies linked to AI that rise to the forefront.
 
AI has taken shape over the past decades thanks to the convergence of a variety of different technologies. ‘Big Data’ requires algorithms to classify, process, and combine large quantities of data and to derive regularities, homologies, and predictions from it. Digital platforms such as 5G or the Internet of Things (IoT) are datamining technologies directly linked to devices of daily use: if watches, cars, or washing machines are now intelligently connected via the internet, it is less for the direct benefit of end-users than to feed databases, which autonomously refine algorithms to make them continuously more “intelligent.” Advanced AI can be summed up by the fact that combinations of thousands of algorithms become "self-learning" through interrelation: they train themselves on databases, partly already their own, with a minimum of human intervention.
 
In recent years, China and Singapore have demonstrated the effectiveness of these technologies for surveillance and population control: facial recognition, real-time monitoring of citizens, cross-checking of citizens’ data (particularly of banking, social, medical, and judicial data) to build a "social credit" rating, codes to restrict citizens’ access to certain places, including locations in the internet. Obviously, these technologies are proving exceptionally useful for authoritarian regimes. What about their use in “liberal” democracies?
 
Quebec - A World Leader in “Civil” and Civic AI
To answer this fundamental question in terms of whether the center of gravity of re-globalization will be inclined to the authoritarian or the liberal-democratic model, Quebec becomes interesting as a case study. In recent years this Canadian province has invested a lot of money1 in AI related to smart cities technologies and lobbying, claiming to be the “Northern Silicon Valley” (in reference to the area between Toronto and Montreal). Among other initiatives, Quebec has created a triple-helix cluster founded on an industry-university-state lab (so-called “Element AI”) in Montreal that brings together world leaders in AI (Facebook, Thales, etc.) thanks to the tutelary figure of University of Montreal professor Yoshua Bengio.2 
 
An international celebrity and a central figure in local and global academic institutions, Bengio has become a media figure in Quebec, sharing his expertise and perspectives on the ethics as well as the medical, educational, legal, and social potentials of AI.3  In March 2020, Bengio declared on public television that he and his collaborators are developing a tracking application based on the Singaporean model to "change people's behavior in times of Corona and make them more aware of the risk they pose to the people they love, but also to society." A few weeks later, his argument became stronger: "There are studies showing that [AI] could work as a kind of vaccine [sic]."4  He repeated to various local, national, and international media that his geo-tracking application was ready and effective, lacking only the legal and governmental authorizations for release. 

covi app marketing graphic

Marketing the "covi" contact-tracing app developed by Yoshua Bengio.

The governments of Quebec and Canada were quick to follow Mr. Bengio’s recommendations by observing what was being done elsewhere in the world. In fact, most governments are interested in knowing and rationally predicting the behavior of their citizens within a legal framework accepted by the population. For example, Amazon.com made its facial recognition software (ReKognition) available almost free to the American police, while Alphabet (Google’s umbrella company) invested more than $50 million to "help" Toronto city become fully smart by taking over the management of an entire neighborhood, Sidewalk. This of course resonates with the modern utopia of total governmentality: to have tools so sophisticated, anywhere and anytime, that they make it possible to understand and know what individuals are doing in real-time, and potentially, where useful or needed, to provide guidance to these behaviors. 
 
In the end, Mr. Bengio’s application was refused by the supervisory governments, local and federal—curiously, without explicit explanation. In the meantime, it has become clear that throughout the world these and similar applications are a fiasco: very few are downloaded except in authoritarian countries where doing so is imposed from above. In Quebec, the backlash provoked by Mr. Bengio’s AI and his statements about it led to significant public debate on the privacy and uses of such an app.5  A public consultation and a parliamentary committee were also set up by the provincial government to measure the degree of public (dis-)acceptance. 
 
Redefining Governmentality Technologies After the COVID-19 Crisis
Bengio’s failure needs to be put into perspective. While the promoters of “universal” AI have lost this battle, the major national debates in France, Germany, Belgium, United Kingdom, Brazil, and elsewhere around tracking-apps have definitively publicized a much greater range of global socio-political issues than just of algorithmic governmentality related to COVID-19.  
 
Technologies specific to tracking-apps are common and already used in many smart cities, on the model of micro-targeting advertisement, framing individuals primarily as consumers with regard to public and municipal services as well. The reframing of public, political, governmental, and administrative support for this type of approach is not new: smart cities projects have existed for a decade. But, the COVID-19 crisis has led some governments to impose these technologies on citizens and in the public sphere, which until then had received very little public discussion. Many citizens contribute to the debate by stating that they are opposed to the use of facial recognition, most recently mixing the controversy over tracing-app with the debates on police violence in the USA.

The debates on Artificial Intelligence as related to infection-tracing applications are representative of [a] transition in forms of exercising power...

Overall, the COVID-19 health crisis is a political opportunity for a redefinition of the relationship between states, citizens, and governmentality technologies: what is needed is debate and policy guidance on tools that govern the behavior of individuals not by rational human decisions but based on algorithmic ones. But this algorithmic rationality is not based on moral principles, political, health, or social objectives: it is a purely instrumental rationality (Max Weber) that calculates, computes, and compiles scattered quantities of data automatically without much human oversight. Such oversight is not desired by the promoters of AI due to the difficulties in understanding the reasons and the decision tree leading to the results of the algorithmic calculations. In essence, the ultra-rational apparatus cannot be rationally understood by its developers. This is what Pasquale calls "black boxes": we know the input and output data, but we have no idea about the interference between the algorithms. Who then can be blamed for this or that decision made by algorithms in a potential future?  
 
Thus, a redefinition and a reframing of governmentality in light of re-globalization (Benedikter and Kofler) is already (and in part, unconsciously) taking place around the world, based on instrumental rationality and inspired by the economic and advertising models of the transnational web giants. The competition between them to be the leader of this new "biopolitical" (Foucault) market is fierce. Smart cities technologies and commodities work as trojan-horses: security, free wifi, traffic regulation, centralization of medical databases (google health), and so forth pretend to offer more efficiency for institutions and citizens, but they also lead to more control.6 
 
Indeed, state institutions do not necessarily have the means—technological, intellectual, and above all financial—to make political decisions with these data. At the birth of modern states (Supiot, Durkheim), statistics held the promise of being an effective tool of governmentality for these new states. In the wake of the COVID-19 crisis, a new form of algorithmic governmentality based on the model of smart cities and digital platforms is taking shape, and it is already reframing globalization. Nobody knows where this may lead, but caution is advised.

Notes

1. As of September 2020, nearly CAD$500 million, excluding tax exemptions.
 
2. Bengio is one of the “godfathers” of contemporary AI together with others such as Yann LeCun and Geoffrey Hinton. He earned the Turing prize in 2019.
 
3. He is the founder of “Element AI” and Mila (a Quebec AI research institute), and takes part in OBVIA—an international observatory for the societal impact of AI and digital technology—supported by the Canadian government to prevent, protect, and educate about the social impact of AI.
 
4. The first statement was made on Radio-Canada on March 30, 2020, the second on Radio-Canada/rdi on April 24, 2020.
 
5. More than 100 articles have been published within 3 months.
 
6. Two examples in Quebec, among others in the whole world, should attract our attention. First, the Quebec government has launched negotiations with private actors, notably Amazon, to host and secure Quebecers’ personal data, arguing that the maintenance of these data by local institutions would be too complex and too costly for taxpayers. Second, the Quebec government hopes to encourage pharmaceutical companies to set up operations in the province by offering them access to Quebecers' medical data.