In June, ChinaFile published a new report, "The Locknet: How China Controls Its Internet and Why It Matters"—the product of 18 months’ work by Jessica Batke, ChinaFile’s senior editor for investigations, and Laura Edelson, assistant professor of computer science at Northeastern University. The report gives a concise but thorough overview of China’s online censorship system, including the motivations behind it and the mechanisms by which it is implemented. It expands on the familiar image of the "Great Firewall" as a perimeter barrier, adopting a broader hydrological metaphor:
The censorship system’s complexity belies the idea that it is merely a wall. The “Great Firewall” suggests an attempt at impermeability, a hard, static barrier running along the border keeping invading information at bay. But the metaphor fails to capture the scale and dynamism of the system as a whole. Beijing doesn’t just stand sentry over its digital borders, it monitors and censors information flows within the country as well. Moreover, the censorship system was never designed to be impermeable. Instead, it uses the most efficient means possible to minimize the quantity of “dangerous” information available to Chinese citizens online. Impermeability is expensive, and not all that much better, from a practical perspective, than simply filtering out the majority of unwanted content.
In practice, online censorship in China functions more like a massive water management system: an amalgamation of canals and locks that regulates what flows, through which particular channels, and at what times. It adjusts to natural rises and drops in volume, though it can be overwhelmed during a flood. Even when the sluices close, it’s not perfectly impermeable; ripples can always slosh over the edge. This can happen in both directions: a gush can surge in from the outside or what was meant to stay in can leak out.
Just as a system of locks and sluices surrounding a man-made lake can regulate the lake’s water level while tides or rivers flow in and out, so China’s online censorship system can ensure the information circulating through the country’s digital spillways mostly conforms to Beijing’s changing whims. The result is a national intranet that links up with the global internet but manages internal information flows according to its own rules. The result is what we have dubbed “the Locknet.” [Source]
The report offers several different levels of detail, beginning with an overall roadmap, an executive summary, and an introductory essay that conveys the report’s main themes. For those wanting more, additional sections focus on the Locknet’s physical context, its implementation by corporate platforms and government entities, efforts to circumvent it, its growing international influence, and its future development. Still greater depth can be found in a technical glossary (also integrated into the main text) and a series of essays introducing technical aspects of the internet itself, Chinese authorities’ methods of censoring it, and their measures to combat circumvention attempts. Another section by Edelson explains aspects of the Chinese context that she found particularly notable as a newcomer to Chinese politics and governance. While this section will help readers less familiar with China make sense of the report, it may also offer seasoned observers insight into their own assumptions about others’ background knowledge of the country.
Batke and Edelson agreed to discuss the report and the thinking behind it, including how they approached making such a broad and tangled topic accessible to a cross-disciplinary range of audiences. The transcript below has been extensively condensed and restructured; a fuller text will be available soon.
CDT: How did the project come about?
Jessica Batke: ChinaFile got approached. Essentially, the Open Technology Fund was aware that current understandings of how the censorship system in China worked needed a little bit of a refresh. They were looking for someone to do that, and somehow, they got pushed in our direction.
The touchstone that I kept going back to was this 2008 article in The Atlantic by James Fallows called The Connection Has Been Reset. It’s wonderful, just wonderful. But that was the only thing I knew of that tried to explain the technical workings of the censorship system to a lay audience. And that was part of the brief [for this project]: this needed to be a technical review of how the system worked, but it had to be understandable to regular people; and [Fallows’ article] was in 2008, and it was four pages long—it’s like we’re in a different universe.
So I was very excited to take this on, and my first thought was, "I can’t do this by myself, because I don’t know anything about computer science." And number two, I really wanted to find someone that had no China background at all. That was really important to me, and that really got validated. If I only had one good idea in my entire life, it was that one.
[…] Laura Edelson: I think about this a lot, because almost all my research is at the border of computer science and something else. I think it is surprisingly rare to have two people from different disciplines participating as equal partners in a piece of work. And what that meant is that we didn’t have to subsume one to the other, and we really were able to generate, I think, novel insights.
JB: The thing that was so awesome for me was that nothing was taken for granted. Laura came in and was like, "if I was building this system from first principles, what are my first principles, and then what does that mean for the technical design of this system?" And so you’re engineering it mentally from the ground up, rather than working backwards, and I think that that made all the difference.
LE: The flip side of that, too: one of the things that is so difficult as a technical person, trying to explain technical systems to lay people, is that so many of the reference materials out there assume a fairly high level of knowledge. They assume that you are in a bachelor’s computer science program. We don’t always have to explain the inner workings of things to a wider audience, and when we had to explain things to each other, it really helped us look at things with fresh eyes.
[…] JB: I actually felt like Laura had a much harder job, because with the things that she was learning, a lot of times there isn’t one right answer, right? The things that I was learning were things like, "This is what a data packet is, and these are the two mechanisms by which they are sent." There is an answer. Sometimes it was poorly written or hard to find, but there is an answer. And that just isn’t the case for a lot of stuff to do with China.
[…] LE: Can I maybe talk a little bit about what I think are opportunities for the future?
CDT: Please do.
LE: We need to be able to talk between communities about where opportunities lie, and what might be fruitful lines of research, productive things that we should be building, given knowledge of both technically speaking, how the internet works and also the broader China context. We need to be able to have that conversation back and forth. So as a matter of strategy, if we think about the weaknesses of a state actor, the primary one that I see is just that they’re slow. They’re big slow bureaucracies.
When I think about building circumvention tools, I think we need a bit of a strategic shift. Instead of making monolith technologies that are really robust and really technically sound, but are single points of failure that a state actor can devote a lot of resources to taking down, we should be investing in smaller, frankly shorter-lived bits of technology that might only stay up for 18 months. That’s okay, if we make enough of them, if we change how we invest in new projects so that we’re making a lot of small bets, rather than one big bet. I think as a strategy that will be more robust against a state-actor adversary.
Something that is really an emergent property of the circumvention ecosystem is the rise of, effectively, circumvention-as-service providers ["airports"], where there’s some guy and you pay him some amount of money per month, and he gives you a box or a hotspot. What it’s using, the specific technology stack, might be changing every month, but that doesn’t matter to you, because that guy is dealing with it. What we should be doing is building a range of tools that can enable those circumvention-provider middlemen to have a range of options, to be nimble, and they’re going to provide the final-mile delivery of customer support and the technical heavy lifting of switching whatever you need to be switching that month to adapt.
JB: The other thing that I’ll say about opportunities for the future: some of them have to do with the fact that the Locknet is not staying just within China, that it’s affecting the global internet and us too. One of the other things that we talked about was increasing transparency for users [outside China]. So for example, we talked about RedNote. It might be helpful for people who download RedNote here in the U.S. to see: "This app is subject to censorship according to Beijing’s rules." That basic level of transparency might be helpful for people, because I don’t know that everybody fully understands that.
Another thing we talked about is making sure people are aware of and sending people to internet standards meetings, because a lot of what’s happening there is going to have a huge impact on how the internet systems of tomorrow function at a technical level, and what it will allow in terms of governments’ ability to surveil or censor if they so choose. A lot of the ways in which bad things could happen are simply because folks who have an interest in maintaining privacy built into those systems are not organizing and making sure that there are people at every single meeting, every single time.
[…] LE: I think this really gets to the point that people like to think about their technical systems as just technical systems that don’t have human values baked in, but they absolutely do. You know, the reason that the internet works for you is because you share the values of the people who made the internet. The internet didn’t work for China’s Party-state. It had big problems with things like the free flow of information, the way that the internet enables people to, at least digitally, have freedom of assembly. And so when it was building the internet [within China], very early in the process, it wanted to make sure it had a pathway to building the Locknet. And you see this in some of the early documents: they wanted to make sure: are we going to be able to get to where we want to go, where we can control the flow of information? Because that is, at least to them, a core value.
CDT: I think this is a good point to segue to the inevitable AI question. How is AI being used so far in the information control system, and what are the prospects for it in the future? Regarding your earlier point about China’s bureaucracy being big and slow, is AI going to help make it more flexible and easier to keep up? Another point I’ve been wondering about is that a lot of the time AI is substituting for human labor, which is not something that’s in short supply in China. So is it possible that, in the Chinese context, it’s actually not going to be such a big game-changer?
JB: It’s important to to understand that there’s two sides to this coin—two ways in which AI is being used, one of which is to augment the human labor of the censors, to identify content that users have produced, and to identify it as problematic and help flag it in the system and get it down. And the other way is to produce content that is already censored [in the process of being generated]. Those are two different functions that AI systems are being used for, and it’s important to disaggregate those. I think in the first case, where they’re augmenting people, we’re already seeing that. But I don’t think there’s any point at which the human gets removed from the loop entirely.
LE: What gets censored isn’t fixed. New things are censored every day, and that means there’s always going to be an important role for humans to play in that system. But I do think that AI is going to make those humans more productive and … I don’t think it’s necessarily going to lower costs so much as it’s going to make it easier to keep up with the growth in content that gets generated.
JB: Chinese companies have this extremely awkward mission in terms of their content moderators, which is to get rid of things that the Chinese Party-state wants memory-holed. Therefore, they have to teach people this information which should not exist, so that they can then attempt to memory-hole it, and that is just an extremely awkward position for them to be in. AI doesn’t completely obviate that, but to the extent that you can program in some of that stuff—and that’s the stuff that’s most likely to be permanent, right? Like the Tiananmen Square Massacre—that does help you, because if you can get rid of 99.999% of that in your first-line AI review, you don’t have to educate actual humans on these things that you want memory-holed.
LE: Getting to the other question: in addition to using AI to execute censorship, AI results and responses are also going to be censored. I think the reason I find this so interesting is that it gets to this larger point that not all technologies are equally easy to censor.
Let’s say I want to censor search results. There are times of the year where particular politicians’ names are censored—you won’t see any results for them. At other times of year, you can search for those people and see results, so you can see that the censorship is taking place. That works for search results because I type in a search term, and I get back some results, and I can see that in this time period, I don’t get back any results, or I get back three, versus the rest of the year, when I get back 10 pages of results.
However, what if, instead of consuming my news by going to a search engine and searching for terms, instead I’m consuming information in a content feed on a social media service? I don’t know what isn’t being upranked in my feed. I don’t know what isn’t being inserted there, because the particular way that that is assembled is hidden—not for any censorship reasons, that’s just the way that that technology is built. And this means that content feeds are easier to censor, and also that that censorship is always covert.
Censorship of AI systems is also covert in this way. There’s all sorts of reasons that people like going to a chatbot and asking a question and getting an answer, as opposed to going to a search engine and giving a search term that’s relevant to their question, and then going and reading the answer … it’s just easier, right? […] And I’m sure it’s obvious why this is an easier surface to censor, and to censor covertly, because the model controls the fact-selection.
CDT: Were there particular misconceptions that you were aiming to correct with the report?
JB: We didn’t come into it, I think, with a ton of things we were trying to correct, other than that the existing model was outdated and oversimplified—I credit Laura for this—because we were coming to it from first principles: what is this model? I actually think that was a much more useful way than probably what I would have personally done, coming to it with knowledge, thinking, "Well, this is wrong, and we need to blow it up." Because we came from the ground up, only when we built up to a certain level did we then think, "Aha! Now I see why this is incorrect, and we need to update understandings." But that happened more organically, if that makes sense.
LE: One of the most serious is that particularly inside the computer science community, we looked at the individual components [of the censorship system]. And when you look at any of those individual components, you might think, "If you want to get around this, you can. If you put in the effort, you can get around this." And maybe you think that’s intentional; maybe you don’t. But I have come to think that this system is much, much more effective than perhaps the dominant narrative would suggest. And I think that change of perspective came because we did take this perspective of looking at the system as a whole.
CDT: And in terms of that effectiveness, the limitations and porosity of the system are a feature, not a bug, right? They’re what makes the system as a whole practical.
LE: Yeah, they’re what makes it efficient. If you think: "I have X number of yuan to spend on controlling the flow of information over the internet, how am I going to do it?" What is the efficient way to spend those yuan? It’s to have systems that are each fairly … not minimal, but not overengineered, but to have a lot of them. That’s a really efficient, adaptable way of building a system, and it’s how we build other kinds of systems—that’s a very normal engineering approach to solving a problem.
JB: Another piece of this is understanding that just because there are gaps, that doesn’t mean the system as a whole is not effective. The human psychological component to this is that for most people, most of the time, as long as it’s inconvenient, that’s enough. Back in the day, people really did want to circumvent, because the outside internet was so much better. That’s not the case anymore: there’s a whole domestic ecosystem that’s really great for all sorts of things. It’s really important to keep this human component in mind. I speak for myself: humans are lazy. I don’t want to do more work than I have to do, and that’s all [the censors] have to do. Just make it good enough for enough of the time and enough of the people.
CDT: I often think about—nothing to do with censorship—studies about how every extra tenth of a second that it takes for a page to load will deter X percentage of users. The tiniest speed bump …
JB: It can put people off. A huge part of this, actually, is literally just that the Chinese government has under-built the actual physical infrastructure that connects the Chinese internet with the outside internet, and so it is slower. A lot of times people will think, "Oh, it’s because it’s being censored. That’s why it’s slow." And that’s not at all what’s happening, it’s literally just that the infrastructure sucks. And that is another one of these ways of introducing friction that has the effect of censorship without having to actually implement censorship. It’s very clever.
LE: I do think it’s worth saying, though, that the outside global internet remains very appealing to people inside China. And also worth remembering that the thing that is appealing is probably Netflix and gambling and pornography, not news in foreign languages. I think this is getting back to where the opportunities are: I think we should be building circumvention tools that work for that commercial market. Because when people go out to get their Netflix and their Marvel movies, keeping the door open for information and knowledge about what is going on outside China, that is still a good and important thing.
CDT: You talk in the report about a shift towards covert censorship. Traditionally, it’s been quite in your face: you know, the notices that say, "In accordance with relevant rules and regulations, this content has been removed” or “can’t be viewed." And the overtness has been part of the system, in that it lets people know that things are off-limits. It helps them keep that in mind. But from what you write, the system seems to be moving away from that.
JB: Yeah, some of this is what Laura already talked about, which is the nature of newer technologies themselves that are inherently more covert—the way feed algorithms work in social media, that’s happening already.
But also, once Google was out of the picture, and more and more global companies were leaving China, there just wasn’t the same pressure to say that "according to relevant regulations, this content can’t be shown." There was no one to push back. No one had any leverage to do so, and so they stopped doing it. As you said, there is a didactic function to overt censorship: you are teaching people what they should and shouldn’t do, and you are, by extension, encouraging self-censorship, and that is the cheapest and most effective method. You do lose that when you shift towards more covert methods. But overall, if you’re trying to shift the mental landscape of internet users, from a very long-term perspective, it’s helpful to you if they think that that’s organically happening, rather than that they’re constantly chafing against restrictions.
LE: The only thing I would add is that I have no idea what the full motivations were behind the real-name ID system. But certainly one of the benefits is that it creates a bit more of this overtness, the signaling to users of the fact that you’re being watched and that you should not violate the rules, because we will track you down. And I think this at least compensates for that loss of overt censorship.
CDT: You highlight the role of WeChat as a way of extending the censorship bubble to the diaspora. Do you have any policy prescriptions for ways to address that?
JB: I think it’s the same thing we were saying earlier, which is transparency. If you are using an app in the United States, it should have to say to you that this is being censored. I’m personally not in the business of telling people that they can’t [use a platform like WeChat]. That’s the main way that people have to communicate with their families back home—there aren’t a lot of other options for them. So if you’re cutting them off from that, what does that do? That’s horrible. So this is a tough answer.
CDT: One of the strengths of the report is that it’s an amazing collection of links to source materials. Are there any that you found particularly valuable, or that you’d particularly like to highlight?
JB: If you want to see how the system has changed since 2008, I recommend everyone to read that James Fallows Atlantic article. And anyone who is interested in how to write, because, God, that guy can write. That’s the first thing that comes to mind. But there’s so much more.
LE: There is an original paper by the father of the Chinese internet that proposes a way to filter out information. I found that paper so illuminating, because it proposed that blocking be proportionate to the likelihood of harm. It proposed, "We’re going to make a probability estimate of how likely this thing is to be bad, and then if it’s like 80% likely to be bad, then we’ll be 80% likely to block it."
And the reason this is such an "aha!" moment to me is that, on a certain level, that is actually how this whole system functions. It’s not how any one component functions that we know of, but it’s very much how the system as a whole functions. It’s very efficient.
CDT: Laura, is there anything that you read about China that you found particularly illuminating?
LE: Oh, there are so many good ones. I think that, to give a really boring answer, "The Search for Modern China" by Jonathan Spence was a very good ground-laying.
Something that really helped me with thinking about a different framework of values and morals was [Jiwei Ci’s] "Moral China in the Age of Reform."
"China’s Thought Management" [edited by Anne-Marie Brady] was also pretty helpful. That was a good one.
CDT: The foreign media presence in China has been decimated in the last few years. On every front, there’s less information getting out of China: fewer official statistics, less independent reporting inside China, less room for academic fieldwork, and so on. A related phenomenon that you write about is the growing difficulty of probing the censorship system from the outside, in terms of "bidirectionality." Can you talk about that now?
JB: Bidirectionality is absolutely crucial for us to understand how the system works. It’s extremely hard to set up a testing infrastructure inside China that accurately reflects what it’s like for a normal person to access the internet. The way we know what’s censored is by throwing packets, essentially, from outside, into China, and seeing what happens, and knowing that you’re getting the same results as if you were inside China. And if that goes away, if bidirectionality goes away, we will lose our technical capacity to monitor at a granular level how exactly these protocols are working, how they’re censoring things, and what is happening. You can lose bidirectionality on some protocols and not others—it’s not a fully binary thing—but it is a really scary thought and a really scary time.
This is another one of these things that has been a bedrock, foundational assumption for decades, for people that have been studying the Chinese internet. I feel it speaks to what a moment of potential crisis that we could be in very soon, which I don’t think we’re prepared for.
LE: I think this actually highlights one of the one of the recommendations that’s normally in our recommendations pitch, but we didn’t mention here, and that is: we need more consistent monitoring of how this system works. Right now, one person will run a study; they’ll collect data for nine months to answer some specific question; and then when that’s over, the data collection is over, and there’ll be large gaps. And sometimes someone will discover something new, we don’t know when it started, and that is actually very important for us to understand, for us to understand how the system is evolving on a technical level, and even just things like what is being censored. That would be really, really useful to have a better view of.
