On October 7, the Writing Machines course hosted Kunle Bello, a machine learning infrastructure engineer at Google, via Zoom, for a conversation about large language models from the perspective of someone who works behind the scenes. Bello discusses how LLMs process language, the role of matrices and computation, and the technical work involved in training and improving models.
The conversation also considers the language we use to describe these systems, particularly the word “intelligence.” Drawing on his experience as an engineer, Bello reflects on the gap between the computational machinery of AI and the seemingly intelligent products people encounter every day. We close with the excitement and anxiety surrounding AI, and how users might develop a more informed relationship with the technology.
Ainehi Edoro: Hello Mr. Bello. Thank you for joining us. As I mentioned, our course looks at AI and large language models from the point of view of the history of writing: not writing as a communicative technology, but writing as a notational technology that works with space. We think about how we can understand certain aspects of how large language models work by looking more closely at how writing works as a technology of arrangement.
We’re glad you’re here this week because we’re looking at aspects of the architecture. We just read a piece that explains what a transformer network is, and in particular how the transformer gives context to tokens. We start with something that is more or less context-independent, and by the time we get to the other side, it has gathered so much information and context that it becomes computable in certain ways.
So one place we could start is, if you were explaining the concept of context, or the transformer network to someone, how would you do it? And why is the transformer such an important part of how a large language model makes meaning?
Kunle Bello: Instead of answering it directly, I will actually step back and give a brief history of computer science. The goal of computer science was essentially to provide the two parts of a language. You can think of the syntactic part and the semantic part. The syntactic part is where you have the grammar and the rules to combine words, and then you have the semantic part.
For a long time in computer science, we’ve done a pretty good job with the syntactic part. There are different types of computer languages that we use for building products and systems. But many of these languages are very heavy on the syntactic part.
The semantic part is essentially unlimited, right? You have all these words, but you can put them together in different ways. And it can even be the same words, but just because you switch them around, they will have a completely new meaning. So that has been a challenge for computing.
What large language models did was essentially make a very good breakthrough in the semantic part.
Even before large language models, you probably heard about some other machine-learning approaches. One of these was reinforcement learning. The idea is that you have a system and an environment, and you let the system, which is essentially like an actor, try different things. Then you can reward it if it does something good. Otherwise, you send a signal that what it did is wrong. If it does this a number of times, over time it can start to achieve different goals.
We made such good progress in reinforcement learning that, for many of the games we know, like chess and Go, we were able to build systems that did very, very well.
But if you think about it, those systems are somewhat closed, in the sense that they have rules and the rules are regular. If you step back and look at things relating to humans, it quickly gets messy, and these systems cannot handle it well.
So what large language models did is tackle that semantic part, the meaning part, directly. They build meaning from the relationship between words.
For example, when we say, “Mary had a little…” most people say “lamb,” right? They won’t say, “Mary had a little goat,” or something like that.
What large language models did was essentially build this massive knowledge database. You can think of it as a table, but unlike a normal table that we build, like a spreadsheet, it’s multidimensional.
The idea is that these words have all these interrelationships, and what the model tries to do is learn these relationships. The ones that are likely to go together have a higher probability than those that don’t.
As it continues to train, it makes guesses that actually make it appear as if it understands what it is doing. So you cannot tell the difference between an LLM and a human, because the answers it gives are very similar to what a human would give.
That’s why we say that it is thinking and that it has meaning. But to be very clear, LLMs don’t think like humans, in the traditional sense.
Because they are pretty good at appearing to understand and deriving meaning from the relationships between words, which are represented as tokens, they essentially look like they have intelligence. And that’s why we call it artificial intelligence.
Ainehi Edoro: You mentioned that LLMs don’t think in the way humans think. But because they perform tasks that look like cognition, we tend to attribute intelligence to them. Do you find the term intelligence helpful for understanding what LLMs do? Or is there another term that better captures what these machines are doing?
Kunle Bello: The way I look at it is that AI, artificial intelligence, sounds good, right? If you say “smart machine,” it doesn’t sound as good as “artificial intelligence.” So I can understand why people go with that term.
Is it intelligence? Can it do a good job of appearing to be intelligent? Yes.
We’ve gotten to the point now where it can play games, it can help you put together your to-do list, it can suggest a plan for your weekend. It can even create images for you, or videos. You can chat with it. So in that way, we can say it’s intelligent.
The challenge is that because we use the same word for humans and these machines, some people do not appear to understand the difference.
For example, an AI system will happily tell you something that is completely wrong with the same confidence as something that is not. That’s what we call hallucination.
If you’re a human, you might say, “Okay, obviously this is stupid. Why did you suggest this?” You have to tell it, “No, that is obviously not right.” And then it will try to correct itself.
Ainehi Edoro: We tend to encounter LLMs from the user side. And like a very good magic trick, they look like intelligence because all we see is what they do for us. But you’re behind the scenes. You’re sort of at the butcher shop, right? You see how these things are built. So I’m curious: what does an LLM look like to you? When you’re working on the computational side, what exactly are you looking at?
Kunle Bello: And that’s why I said that it’s not really intelligence, because I know how it works. I like the example of a butcher shop. The way a butcher looks at the product they work on is very different from a restaurant that gives you this nice meal.
From my side, it’s a little bit uglier and it’s a little bit messier, because I’m not seeing the final product. From my perspective, I see it as a very well-designed, coordinated system, with things working together to appear to be intelligent.
Essentially, think of an LLM as a very large matrix. You can think of a matrix as just some numbers that you arrange in some kind of dimension, like two by two or four by four. Just numbers that you arrange.
It has these layers, and each of these layers is made up of matrices. But what makes it impressive is that instead of having a five-by-five matrix that you can write in your notebook, itt’s really huge.
Ainehi Edoro: Can you help us imagine that scale? We read about high-dimensional vector spaces, sometimes involving thousands of dimensions. I know it’s not something we can easily picture, but can you give us a way to think about it? What does it mean to work with a matrix of that size?
Kunle Bello: Maybe I can try to use an analogy to explain it. If you’re a physicist and you’re trying to compute force, force is equal to mass times acceleration. But because we live in a 3-dimensional space, Mass would be a certain dimensional matrix, and acceleration would be another dimensional matrix. Matrices have a certain kind of mod— sorry, I’m getting a little bit technical.
Ainehi Edoro: That’s okay.
Kunle Bello: I’ll quickly go over the details
So you have these two matrices representing mass and acceleration. When you say you compute force, what you’re literally doing is combining them using the matrix multiplication rule.
You say the first row is multiplied by the first column of the second matrix, and then you sum it. Then you do the same thing, the second row multiplying the first. Eventually, you get a single number, and that number is the force.
From a physicist’s perspective, when you say, “I computed the force,” you just multiplied some matrices. You can also do it in college. You can just do that matrix and then do the computation.
Now, instead of a 3-dimensional matrix, suppose it is a 100-dimensional matrix. It’s still force that you’re computing, but now the pieces of the mass that you consider are from a 100-dimensional space.
It’s almost like you take something and break it into different parts. If you look at a human, we have different parts. You can consider a human as a single human, or you can think of all the organs, all the parts of the human body, connecting together to form a human.
And then you can still go down further. The eyes—you can think of the iris, and you can still go down like that.
You can keep breaking it down. Something that is one can become the head, limbs, chest, and it’s now ten things. Then you can blow it up to a hundred. You can blow it up to a thousand, to represent a human. In fact you can get all the way to the atoms Now you’re talking about trillions of atoms.
It’s the same thing here too. What you’re trying to derive is that if this property and this property and this property interact, what do you get? You want some value to capture that. But you want the properties to be as real as possible. So you expand it into these vast dimensions, and then you compute it.
But the point is that it’s still the same thing. If you want to get a value, you get two things that you’re connecting together. You’re looking at the interaction. It could have been force equals mass times acceleration, which is just a small dimensional matrix, or it have entries that run into the billions or even more. You still want to get a value to represent the interaction. Hopefully I didn’t get too technical. But the key thing here is that the dimension is just to help you get to the little facets and details. The action itself is still just matrix multiplication.
H. Z. (Student): In the article we read this week, transformer networks are layers of matrices, and you multiply them by some vector, which is your input encoded into a vector.
For models, we update them every few months. For example, we have new versions of GPT and new versions of Gemini. For those new models, what is updated? Do we change their structures, or do we just add more layers and then train them with a larger dataset, and maybe with more experts?
Kunle Bello: We actually do both. The way to think of a model is in different parts. There is the model itself, the actual model. Then there is the data. And then there is the training. When you want to do a new training exercise, you may have to change the model. There’s this thing called weights. Remember, I was talking about how, for each of the entities, you can either consider them as one, or a small matrix, or a large matrix. The weight is what helps you assign preference to the pieces that make up that matrix.
When you do your research on the model, you may realize that you need to tweak the weight a little bit. You have to reassign the weights because of some data that you found that would help improve the model. That’s the model-enhancement part.
Then there is the data-enhancement part, where maybe when you did the first training, the data that you used was not partitioned in a way that would have brought out the best improvement in the model.
The way we do training is that you have a set of data you want to use. You might break it into 80 percent and 20 percent. You train the model with 80 percent, and then after you stop, you use the 20 percent to check that it does not get overfitted.
In training, you want to make the model able to answer questions correctly. But you don’t want it to only answer questions for the data that you gave it. That’s what’s called overfitting.
When you’re getting this data, sometimes the way the data is [organized] has certain patterns. So the way you shuffle the data to reduce overfitting and improve the ability of the model to get better results is what’s called data engineering. There are people who are good at trying to massage the data to make it better.
I’ve talked about two things now. There’s the model itself, changing the weights. There’s data engineering. And then there’s what is called the infrastructure that you build. Because these things are so massive, you have a set of computers, networks, and things like that. When you’re doing the training, some parts may not be pursued because they take too much time. But if you improve the network, you might actually be able to take that path. And the data you get from accessing that remote node may actually improve the overall model. So there are many pieces that can change when you want to do another training exercise. Anytime they say there’s a new model coming, it’s usually a combination of these three things.
Ainehi Edoro: Thank you very much. We’re almost out of time, but I’d like to close with one last question. Your world seems to me to be very far from the madding crowd of AI headlines and public discourse on the technology. You’re working directly with these machines, while outside the engineering world, there is chaos over the fears and promises about this technology. I’m curious whether working behind the scenes gives you a different perspective on all of this. What would you say to people who are trying to make sense of AI amid all these competing claims about its future?
Kunle Bello: That’s a good question. I always suggest that you should try to understand things in all these different facets. From the user perspective, what you’re sold is this impressive innovation. And there’s also this desire to push the limits in terms of what this system can do. Some of it may border on fearmongering. People might think, “If AI is going to do everything, then maybe I don’t need to go to college or even specialize in this area, because AI would do a much better job.”
Now that I’ve given you some perspective from the inside, you realize that it’s still software. It’s still computing. Those are the two views. But what you could do is just look at things as they are. Instead of worrying about how AI could take away jobs, or AI could cause a Third World War, look at it just as a system. You have a phone in your hands, and you use it. You have computers at home or in the library, and you use them. You can use AI like that.
You can say, “Okay, if I’m going to do something, this is a useful tool. How can I use it in a way that will make me better off?” If you just offload your critical thinking to AI, then you will not be different from the masses, because you’re just following what AI tells you. But like every tool, you want to keep your critical thinking. You want to challenge it. And sometimes you actually want to take some time to dig into an area before you ask AI, so that you have something to validate it against.
If you do that kind of exercise, that would be a productive way of using AI, because then you can see when it’s hallucinating or when you think the solution is not the right one. Together, you can come to a much better outcome.
I always encourage people to take the middle ground, because AI is not going away. Just like computers never went away, even though some people felt they were going to completely change the world and some people would lose their jobs.
AI is also not going away, but it’s a tool that you can use to be more productive.
Ainehi Edoro: Thank you so much for joining us, Kunle. We really appreciate it.
Students: Thank you!
Kunle Bello: Thank you very much.
Kunle Bello: Mr. Kunle Belo grew up in Ajegunle, one of the poorest neighborhoods in Lagos, Nigeria. He taught himself through high school curriculum and studied Computer Science from the University of Benin, where he graduated with first class (summa cum laude). Mr. Belo worked for a few years in Schlumberger, Nigeria before moving to the United States, where he did a stint in the Maths PhD program in Ohio state. Deciding the degree will take too long, he left the program to join IBM in Austin and in 2016 moved to Google as a Staff Software Engineer, where he built the scheduling infrastructure to run machine learning workloads including, Gemini.
Image: “Composition” (1949) by Lee Kransner
Image source: The New York Times
