From Orphanage to Google Brain: David Sussillo on Heroin, Neural Networks and the Mysteries of the Heart

Andrew Keen (Host): Hello everybody, my guest today is a distinguished neural reverse engineer, a very influential figure in Silicon Valley. He’s worked at Google Brain Labs, now he’s at Meta Reality Labs, and he teaches at Stanford. And he has a new book out; it’s his autobiography, it’s called Emergent. But it’s a different kind of story. Usually in these kind of books, our scientists, our successful scientists go from distinguished school to distinguished school. But our guest today, David Sussillo—his narrative is quite different. He attended two schools which most people haven’t heard of; I certainly hadn’t. One called Albuquerque Christian Children's Home and another called Milton Hershey School. Both for, shall we say, underprivileged or unfortunate students or kids. And it’s an interesting story. David Sussillo is joining us from New Mexico. David, is that a fair way of putting it? Is that the unusual thing, the interesting thing about your memoir, Emergent—of your unusual background, maybe not so much unusual background, but your background through your family and then your narrative of going to all these fancy schools, getting a PhD at Columbia, teaching at Stanford, working at all the top labs in Silicon Valley?

David Sussillo: I think you got it. Thank you for having me on, by the way. I appreciate it; it’s an honor. Yeah, I think that’s right. The sad truth is that most people who start with the kind of background that I started with, or that my sister started with, or indeed many of the kids that I lived with started with, do not fare so well in life. So, you know, the Albuquerque Christian Children's Home is a modern-day orphanage; it’s a group home, right? It’s a place of last resort for children. And the Milton Hershey School, of course, is associated with Hershey chocolate. The founder of Hershey chocolate, Milton Hershey—the story goes—he and his wife couldn’t have children, so back in 1906 they donated their entire fortune to an orphanage that they started. And so that school is very different from the ACCH, which is just different orders of magnitude. But yeah, those were the places I grew up largely.

Andrew Keen: I know your book took you five years to write. You tweeted about getting the book and your excitement. Probably took you a lot longer than some of your computer achievements. Why did you write this book? Is it a message to other kids who perhaps had your ill fortune as a child—maybe it’s a wrong way of putting it—or was it a way of, was it a kind of cathartic exercise, something you had to get out of yourself?

David Sussillo: Yeah, you know, by the time I wrote this book, I had been in psychotherapy for 20-plus years. I have to say, I wrote the book for the simple reason that I thought it would be a great story. And as I was doing it, it became clear that it was, hopefully, a great story, but also, you know, a marker of what life was like in those kinds of institutions in the 80s. Potentially, you know, maybe somebody would read this and find inspiration in a way that would help them. Maybe people who didn’t grow up in that type of environment could find hints or directions for how to help people. So, you know, all of those things are, I hope, really positive outcomes from writing this book. And, you know, for all that, it was cathartic to write it, even though again I didn’t really set out to do it that way.

Andrew Keen: And by the way, the book is called Emergent; it’s out this week. David, we probably put the horse before the cart here, or the cart before the horse. Tell us the story briefly; we don’t want to give away everything, we want people to read the book, buy the book, read your story. But very briefly, tell me particularly your origins and why your story is so unusual.

David Sussillo: Yeah, so I was born and raised until I was a teenager in Albuquerque, New Mexico, and my parents were both heroin addicts. There was real dysfunction in that family. My parents were basically in and out of different rehab programs and so on and so forth, and there was ultimately a divorce when I was like five or something, maybe six. And so it became pretty clear that my mother, who now is saddled with raising two kids alone, isn’t really working and has mental health problems and a drug problem—that that just wasn't really going to fly. So what that meant for my sister and I is that ultimately that fell apart and I ended up in this group home. So that’s the sort of the background story there. I would say that when, you know, the other component of this is that these—just as a quick outset, I’m a fan of the mission of those institutions; I’m a fan of the mission of the Milton Hershey School. But, you know, to be in one of those experiences is not always great.

Andrew Keen: And I mean even, for people listening, I’m going to the website of Milton Hershey School. I mean this is, websites are marketing, they show their schools in the best light. This school looks rather bleak, even with its website—it's got snow all around. I presume sometimes the sun comes out. David, tell me a little bit more about your parents, their background and how they became heroin addicts.

David Sussillo: Sure, so that is a funny picture for Milton Hershey School for the record.

Andrew Keen: We need to tell the marketing people that too; not take the photos during the bleak winter when it's covered in snow.

David Sussillo: That’s funny, made me laugh. So, so I don’t really know what happened to my mother. You know, by all accounts she would play tennis as a teenager, she played the piano, then something happened midway through high school and she ran away and, you know, we can speculate but nobody in the family knows.

Andrew Keen: You never asked her?

David Sussillo: She’s since passed away and I was—she passed away when I was young, so I didn’t have the wherewithal.

Andrew Keen: And did you know the broader family? Did you have any access to grandparents or uncles or aunts?

David Sussillo: Yeah, I asked her brother and he basically was like, "I don’t know what happened; I was already out of the house by the time she was in high school".

Andrew Keen: What about your father?

David Sussillo: With my father, that was, we know a lot more there. In that case, he’s coming from generational trauma. His father was a captain in the, I believe, the Air Force or the Navy; anyways, he was killed in a plane accident over Phoenix. The whole family was living in Tucson. And so, now practicing Catholics, five kids, all of a sudden he’s passed away—is my grandfather—and my grandmother moves the entire family back to Brooklyn where she grew up. And they basically lived on a pension, very, very poor, with a single mother raising five kids, which just sounds really, really tough. And, you know, my—the other thing is my father’s of a certain age that—like I grew up knowing that certain drugs were kind of okay and certain drugs were like really dangerous. Okay, maybe marijuana—again I’m inserting my own ideas here—but anyways, not okay: cocaine, heroin. I don’t think my father had access to that kind of information. He was—so I guess he was born in ‘50; he was addicted to heroin by 15. So, you know, there was just kind of, he never really made it out of the gate.

Andrew Keen: What was his fate? I mean, you say he was a heroin addict; did he work?

David Sussillo: He, he worked on and off. The best way to view my father is as an untrained preacher. He was religious in the extreme. And so, you know, he would—is really kind of an interesting guy, even though he sort of failed me as a father—he was on the streets of primarily Albuquerque, sort of "preaching" is a little bit funny but, like, you know, having a ministry with other homeless people or drug addicts on the street of Albuquerque. And that was really how he spent his life, combined with managing what I believe was a very real and lifelong heroin habit.

Andrew Keen: Which of your parents had more of an impact on you?

David Sussillo: Ooh, that is a fabulous question. So, you know, one—I’m fairly certain that both of my parents loved me and loved my sister, right? So there was this sort of core of emotional situating that I believe happened to me in my first four or five years. So I guess I would say give equal credit. I mean, at the end of the day, my mother was the one mostly taking care of us after my father split, but that would be my answer to your question.

Andrew Keen: Do you feel and—as I said, you’re a neural engineer, you’re a distinguished scientist, you teach at Stanford—did your parents, did what did you inherit from them? And was it a kind of negative inheritance? Some people look at their kids—at their parents and think, "I want to be, obviously," you know, if you’ve got heroin-addicted parents, it’s unlikely you want to be like them. But so I presume you learned some negative lessons, but do you feel you inherited some of their characteristics, their traits? You’re a scientist, so presumably as your parents, they passed something on.

David Sussillo: Yeah, that’s an interesting question. So my mother’s father was the chief engineer of a milk plant in Albuquerque for the entirety of his career; he was uneducated, I don’t think he finished high school, but he was a smart guy. You know, I think the real component of any sort of genetic inheritance that I have is really almost the entirety of my father’s family are people people—psychiatrists, psychologists, this kind of thing. And I think having some of those abilities in me innately was really, really useful in navigating some of the group home life that I had to navigate through, you know, ages 7 through 18.

Andrew Keen: You wear these group homes on your sleeve, in your bio for the book; you lead by saying that you spent five years in the Albuquerque Christian Children's Home and four years at the Milton Hershey School. Do you feel that they enabled you to go from this broken family, failed parents, to becoming this distinguished scientist, or did you do it in spite of the Albuquerque Christian School and Milton Hershey?

David Sussillo: Can the answer be both?

Andrew Keen: It could always be both, David; that’s what I would expect.

David Sussillo: Yeah, so the answer is both, you know. It comes up like, you know, am I an example—I’ll answer your question but I’m going to back out a little bit first—you know, am I an example of the American Dream, for example? And the answer is yes and no, right? There—I’ll never forget a buddy of mine during my postdoc years; he’s from Albania, right? And we’re out on the beach in California and what is clearly a group home is out on the beach playing; you can tell because it’s mixed race and mixed age and mixed gender and the whole thing. And he could tell I was affected by the experience because I’m remembering back to my days, and he just looks at me and says, "You know, this just wouldn’t have happened in Albania." So these—you know, these are his words, I don’t know anything about Albania—but it just crystallizes for me the idea that there are institutions in place here that, you know, if had they not been there, I would have been in a, you know, very, very dire straits, like on the street-level dire straits. But for all that, you know, those institutions, primarily through neglect, were doing at least me personally a disservice, and if you look at some of the people who I grew up with, the outcomes are mixed.

Andrew Keen: You were one of the people featured in Cade Metz's Genius Makers: The Mavericks Who Brought AI to Google, Facebook, and the World. I’m not sure if you would consider yourself a genius; you’re clearly very smart.

David Sussillo: No, definitely not. Am I in that book? I’m not sure I am in that book.

Andrew Keen: Ah, well that’s my understanding is that you were featured in the book, but anyway, you know people who were featured in the book like Geoffrey Hinton.

David Sussillo: That’s for sure, yeah. Fifteen feet from fame.

Andrew Keen: At what point in Albuquerque, at that school, or Milton Hershey School—at what point did you realize that you had unusual talent?

David Sussillo: Yeah, this was a major moment for me. I was, I think I was in third grade, and you know, I don’t know if it was from the report cards in school or communication between the house parents at the ACCH and the teachers at school, but one way or the other, somebody decided I should get tested for the gifted and talented program. The gifted and talented program being a sort of a program for, you know, different classes a couple times a week, that kind of thing. And so that was an—so I tested and I got the—I made the grade; I got in. And so, you know, that had two major outcomes. Number one is all of a sudden for some period of the week, I’m actually in different classes. I’m not—I had major behavioral problems as a kid—so I’m not irritating my teacher as much, you know, I’m with kids who are more like me, whatever that means. And I’m learning, right? And the other thing was that it gave me a story about myself, right? And so through a lot of what happened through my childhood and teenage years, I really leaned on that story. It really became my identity in a way that, you know, given the alternative is probably a great outcome, but was a little bit one-dimensional.

Andrew Keen: As I said, you’re a neural reverse engineer, so you’re interested in neuroscience. You ended up at one point at Google DeepMind, you worked for Google Brain with Jeffrey Hinton, who’s been on the show—the group that did all the LLM work before OpenAI took it over. How does your understanding of neuroscience, your lifetime commitment to studying neuroscience—how does this connect with the story of your childhood? Are they in parallel? Are they mixed up together? In other words, does becoming a neuroscientist—does that help you or at least in your mind, does that help you understand yourself, your childhood?

David Sussillo: I’ll be honest with you: only a little bit. Pardon me, so they are related. For me when I was in my mid-20s, I was thinking about what do I want to do with my life and, you know, I’d been coding, doing software, and that was cool, but you know, you have this moment like, "What am I really going to do with my life?" And so for me, neuroscience was a means of translating my very specific set of technical gifts into a human domain that could potentially have a positive outcome on the world. So in that sense they’re very much tied together. Now, the big dirty secret of neuroscience is that we don’t really understand much in the ways that people would love us to understand, and I am sad to say I am no different. So there is a little bit of how I think about things based on my training, but largely I think about my background as most people would, through the lens of story, through the lens of psychotherapy, through the lens of relationships and things like that.

Andrew Keen: Why—you mentioned earlier that you spent a lot of your life in therapy—why, David?

David Sussillo: So, when I basically got to college at Carnegie Mellon and everyone else was like really quite prepared—

Andrew Keen: You went to Carnegie Mellon, another very distinguished college, which in itself must have been quite an achievement.

David Sussillo: Holy moly, I got there and I didn’t know my ass from my elbow and everyone else was like really quite prepared.

Andrew Keen: So you came straight—sorry to jump in here—you came straight from Milton Hershey to Carnegie Mellon?

David Sussillo: That’s right, yes.

Andrew Keen: So that’s quite a jump, isn’t it?

David Sussillo: Yeah, you know, I just wasn't really prepared. But you know, it really turned out that I was meant to be there; there was a few hang-ups at first but I eventually found my footing. So, you know, college really has quite a bit, especially American universities—they have all this set up for the kids so that they just enter into a large group of students and they make friends and there’s orientations and there’s all these things to do. And so when that was over, when I left university, is really when I like all of a sudden didn’t know what to do with myself. My whole life had been like this moment, like: educate myself to get out of these crazy circumstances, and then all of a sudden that’s effectively done and I didn’t really know what to do. So, you know, I’m 23, 24, my life really falls apart and it’s because of that that I get into psychotherapy because I was having panic attacks, it was just a terrible, terrible time. And what, you know, sort of I always wanted to be normal, you know, I wanted to be okay, I wanted to be—God forbid—successful. And so I didn’t have any like hang-ups about, "Oh, therapy is this or therapy is that". I wanted to get better, I wanted to feel good, and so when I got into therapy is really when things started going my way because as it turned out—I didn’t know it at the time, you know, it was one of those like lucky crises kind of thing—I would have had to do quite a bit of work in order to address some of the trauma in my childhood, and I was able luckily to do that very early in my adult life.

Andrew Keen: And how did computation play in this? The subtitle of your new book, your memoir: Emergent: A Memoir of Boyhood, Computation, and the Mysteries of the Mind. You studied computation and the mysteries of the mind; how did that play in to all of this?

David Sussillo: Yeah, sure, so I’d always been interested in—you know, I was born in 1975, so computers, like the personal computer, I was like right on the crest of a historical wave. Personal computers didn’t exist really before I was four or five, and all of a sudden all of these technologies—of course I couldn’t afford them, but they were made available in class here or there, or video games for example—they were the thing I grew up with. And so I’d always been interested in the technology and that translated as I grew older into an interest in what even is a computer, how does that work, what is that. Or John Conway’s Game of Life, this type of simulation where all these weird things happen based off of simple rules, as a type of example of emergence. I got into university and was like, "You know, I’m going to study this stuff; this is really neat and interesting". So then to—the idea, I’ll tie it back to the book, which was the source of your question—the idea of emergence is that simple pieces combine to make much more complicated outcomes. And when I think about my own life, I think of it like a crazy quilt. There’s all these just these pieces; like there’s no one can say, "Well this was the thing, this there’s no it’s not simple". And so what I did in the book was sort of tell two stories at once. One is the story of my own life and where I end up and where I came from, and the other is the story of how we ended up in this moment for neuroscience and also for artificial intelligence and basically how AI started from neuroscience. And I tell it through the form of emergence, through the story of emergence, in the hope that by braiding those two narratives together they sort of enrich each other a little bit.

Andrew Keen: Does AI, in your role in the industry—as I said you now work at Reality Labs, you’ve worked at Google Brain, so you’ve been around some of the biggest, most influential companies in Silicon Valley—has that helped you understand your own mind? You the subtitle of the book is The Mysteries of the Mind. Or do you think AI is just digging a bigger hole for all of us as humans, outsourcing not just our intelligence but maybe even our emotions?

David Sussillo: I, you know, just to tackle that in reverse order, I’m very much an AI optimist. I don’t want to be a Pollyanna, but I see just huge capacity for great things when humans are paired with AIs. I know there’s going to be disruption, potentially very significant disruption; I don’t want to make light of it. So but just to give you, you know, if you have any other questions we can go into it, but I’m positive on these technologies. So to answer the other part of the question: yes, a little bit, right? So we don’t know how the brain works, but one of the things that has been very successful over the last decade is using neural network models. Now AI is a big thing with language and it answers questions, but it’s all underneath it are neural network models, and those neural network models really have helped us understand, at least in hypothesis—we’re still working on laying down the experiments in brains—has helped us understand, at least hypothetically, how brains work. So there is a strong connection there. Again, whether or not it helps me understand the precise details of my own life, that’s a little bit of a different question.

Andrew Keen: Do you think your understanding of neural networks might have helped improve your parents' life?

David Sussillo: You know, it’s an interesting—in some, so basically there’s understanding and then there’s intervention, right? I think we’re getting more understanding, but the intervention that is through therapeutics, I think that’s going to come. I would say that a big fat "maybe," right? There’s just a lot of stuff to be done. But I think if I, let me try a different tack: if I look at some of the things we’re discovering now about maybe how brains work and when somebody were to go in and try to turn that into some kind of translational methodologies, then it’s entirely possible that it could help them, yes. There’s a lot of "ifs" here; there’s many "ifs" here but fundamentally, yes.

Andrew Keen: Well you’re a man of "ifs," that’s what the book is about. David, what are you suggesting in this book in terms of kids who have your misfortune of ending up in a place like the Albuquerque Christian Children's Home? What changes would you like to see? Because there are many kids like you; some of them may even be watching.

David Sussillo: Yeah, so the first thing that comes to mind is structural. So somewhere along—I’m not an expert in these things, just full disclosure—but somewhere in the last 40 or 50 years, someone figured out that like the orphanage thing was a very, very bad idea. And so these days, most kids are placed in the foster care where they’re trying to reproduce the sort of family unit and, you know, the thing that I am very much on board with because I didn’t have that myself was sort of a, you know, one set of parents to like one or two kids, where that—because you know really the thing that I contended with was neglect. At the ACCH, depending on how many kids were there at any given time, it was like 10 to 16 kids per house parent, parents, couple. And it was the same way at Milton Hershey School. So to me structurally, the first thing that either of these institutions could do—and I actually think the ACCH has done this—so much more equal house parent to child ratio; in fact I think there’s only four or five kids in a home now at the ACCH. Even Milton Hershey School has, I think, gone from 16 kids, which was pretty crazy, to 12 which, you know, given how much money they have, one suggestion I might make is to reduce that even further. So I would point to that.

Andrew Keen: David, you talked earlier about your experience on the beach in California with your Albanian friend who looked at a group of kids from an orphanage playing on the beach and saying, "Well that wouldn't happen in Albania." But there are orphanages in Albania, aren't there?

David Sussillo: There are, again I’m just going on his take there. Basically what he was trying to say was that—my read on what he was trying to say—is that everything is very, you help your family there, right?

Andrew Keen: Oh I see, so you would have ended up with an aunt or an uncle or a grandparent.

David Sussillo: Yeah, you help your family and that’s kind of how it would work. And so you could now then turn that and say, "Well, these days there’s a very strong push to keep parents with the family," which I’m largely on board with, in fact almost completely on board with. So I think that was the difference there.

Andrew Keen: David, your childhood of course as you've noted earlier was destroyed by drugs, by the heroin addiction of both your parents. Silicon Valley has a history of flirtation with certain kind of drugs, psychedelics in particular. Do you have strong views on that, on this somewhat liberal attitude towards legal or sometimes even illegal drugs in Silicon Valley, very much associated with the computer industry? Steve Jobs for example famously was into them himself, even spent some time in India.

David Sussillo: I don’t have strong opinions. I think I am unaware of like, you know, the titans of Silicon Valley going back and doing lines in the bathroom—that may be true, but I’ve never been exposed to it. I think what you’re really talking about is, you know, psychedelic exploration which, having done my share of that in my early 20s, just feels like a different thing, right? I, you know, those drugs were not addictive to me when I experimented with them. So, you know, more broadly if you’re asking like, how do I feel, you know, just generally speaking in and around drugs, they mostly make me anxious; I don’t really like being around drugs. But in terms of policy, I mean, it seems like both extremes seem nuts, you know; you don’t want people running around with needles and neither do—neither can you stop people from doing it. So there’s this sub-middle road there.

Andrew Keen: I know you've been married 17 years to your wife. Do you have children?

David Sussillo: I do not have children. We thought about it, you know, the answer there I—if Robin had wanted kids, I would have done it. But we were just both sort of ambivalent about it. I think if we had done it just to run that experiment, I won't lie, I think it would have been very difficult for me. You know, all my mentors in life from my aunts and uncles to my PhD advisor to my therapist, they’re all like, "No, no Dave, you know, having kids is extremely rewarding and you would have done great at it". Myself, I think I would have relived a lot of trauma that would have been pretty hard to reckon with.

Andrew Keen: Do you think it would have been harder than writing a book? Took you five years to write the book.

David Sussillo: Yes, I think it would have been harder. You see kids, you relive things through a very specific way of relating. Writing the book, I could do it on my own time; I need a break, I take a break. Kids, you’re not afforded that luxury.

Andrew Keen: Your book—the subtitle is A Memoir of Boyhood, Computation, and the Mysteries of the Mind. As you noted earlier, there was a cathartic element to the writing of the book; you were thrilled when it got done. What are still the mysteries of your life that writing this book didn't answer, that you still can't figure out?

David Sussillo: Ah, I mean sort of giving away the thing, but I mean, I don’t know, you know, I sort of set the framing as like, "Can we even understand how I made it or how people make it generally?" And the short answer is I couldn’t figure it out. I really, you know, it really is a patchwork; it really is a set of circumstances that feel mostly for the most part completely random. And I can point to things, don’t get me wrong; I can say, "Oh, you know, my aunt and uncle’s intervention when I was nine was a huge deal and meeting my first best friend allowed me to make informed intimate relationships and my first love in college was a big deal". So, you know, I can point to those things and they matter, they’re pieces of the quilt if you will. But you know, is that a systemic explanation, is that something that could be potentially applied to other circumstances or to other children? I think there the answer is a little less clear. I mean, surely people need love and all of that, but then there’s this risk of just devolving into platitude.

Andrew Keen: The H-word isn't in the subtitle or the book: Memoir of Boyhood, Computation, and the Mysteries of the Mind. Happiness, David—has this life of yours—it’s an interesting life—have you been happy? Are you striving for happiness? What are your goals?

David Sussillo: Yeah, that’s a great question. I would say I’m happy. You know, it depends, is there true happiness? I mean most people will tell you no chance and I have my own fair share of struggles. But I have found peace with my past; I really have, that’s not bullshit. I am happy with the work that I do; I am in the moment when I’m doing research, you know. I’m in the moment quite a lot. And I still carry around baggage from my childhood, I think we all do, I just happen to have a hell of a lot more than most people. I’ve just learned how to manage that. But I mean in the short answer to your question, I would say yes, I’m happy.

Andrew Keen: You're not an old man, final question David. The book is called Emergent; I'm not sure if you've quite emerged, maybe "emerging" would be an equal title for the book. What do you still want to do with your life? I don't mean in terms of making money or working at a tech company, but what haven't you done that you would still like to achieve one way or the other, maybe in Emergent 2.0?

David Sussillo: That’s a great question, I’m thinking about that a lot. I think, you know, for me the thrill of discovery has always been one of my very favorite things. I think this moment with AI is something that I really want to dive into. I think, so that’s sort of an intellectual curiosity and happiness. I think, you know, if there are ways for me to translate this particular book into ways to help other people or kids, I think that would be a fabulous outcome. Again I didn’t set out to do that, but now that we’re here, you know, I think that growth opportunity for me would be very cool. And so these would be the couple of answers that I have immediately to your question.

Andrew Keen: Well there you have it, it's out this week, Emergent by David Sussillo: A Memoir of Boyhood, Computation, and the Mysteries of Mind. A very unusual memoir, very revealing. David, thank you so much for coming on the show, writing the book, and being so open to discussing a very unusual childhood.

David Sussillo: Of course, and thank you for having me, Andrew, I appreciate it. Thank you very much.

Andrew Keen: Hi, this is Andrew again. Thank you so much for listening or watching the show. If you enjoyed it, please subscribe... Substack, YouTube, Apple, Spotify, all the platforms.