Replit: A Startup Story
You can learn more about Contrary Research and our repository of private company research here!
Full Video
We had the chance to sit down with Amjad Masad, the CEO of Replit, to talk about the story of how Replit came to be, as well as topics like building with open source AI, managing a team with intense flexibility, bounties for budding software builders, and Replit's vision to bring the next billion developers online.
Five Key Takeaways
Pivotal moments that led to the founding of Replit: Amjad’s childhood moments in Jordan. Amjad had always been obsessed with computers from a young age. He started coding by the age of 13. He didn't have computers when going to university. He grew up in an environment where computers were expensive. He had to improvise and overcome obstacles in learning to code. By the age of 15, he started to build software and make money from it.
Code-assisted AI programming is a trend that is here to stay: Replit’s ghostwriter and other platforms like Github’s Co-Pilot will significantly optimize developers’ experience to become more productive.
The most successful people in the future will be individuals that can think well: With LLMs like ChatGPT or AI-Assisted platforms, computers will become good at programming. The next evolution of technology will depend on human’s ability to think more deeply and creatively. It will depend on people’s ability to best utilize these platforms to execute tasks or create applications toward their desired needs.
The long-term vision of Replit is to empower more people to create code easily and build applications on any device while being able to earn something from it: Today, Amjad believes that Replit has only accomplished the first part, but he sees the future trending toward companies providing the AI and monetization engine for optimizing the developer's experience.
Replit utilizes a team culture of decentralization and autonomy: They have more of a loose and self-organizing structure than a strict hierarchy. Every project has a DRI (directly responsible individual) who is responsible for all decisions and accountable. Amjad believes “The thing you have to worry about in organizations is committees. Humans love to create committees for whatever reason. When you see a committee, you have to kill it immediately.” At Replit, they measure progress week-over-week ensuring that the business is making progress on a weekly basis.
Sections
13.25: Articulating the vision for Replit
22.09: How to increase the democratization of software
28.16: Replit’s Ghostwriter and the role of open-source in AI
38.46: Replit’s culture of decentralization and giving people autonomy
46.29: Q&As - Why Replit doesn’t use GPT for training models
48.49: Q&As - AI-Assisted programming and Co-pilot
52.07: Q&As - How to validate a new product with customers
53.58: Q&As - What is Amjad currently learning and reading
Full Transcripts
Amjad’s Founding Journey For Replit
Kyle:
2.40: I wanted to frame the question by talking about your experiences, and maybe you can touch on the highlights of getting to where you are today. What were the biggest things when you think about the most pivotal points when you had to make a decision or what led you to Replit? What was sort of the big pivotal points as you think about your journey to where you are today?
Amjad:
The first time I saw a computer would count as one was in 1993. I remember my father putting together this. So this is Amman, Jordan 1993. I've never seen a computer. My father is an engineer for the government in Amman. He was fairly technical and liked technology. He was really into computers. I saw him plow his life savings into this machine. I remember one of my earliest memories, just remembering him putting together this computer. The moment he turned it on, I was really intrigued. And at the time, we didn't have UI, we didn't have graphical user interfaces. Windows 95 was the first kind of mainstream GUI as we used to call them. And it was a DOS interface. It was actually a kind of a REPL. So replit comes from the term REPL, which means ‘read–eval–print loop.’ So it is the stages of a program to bootstrap a programming environment.
The computer reads what the human has to say or writes eval, and the computer evaluates. So meaning it processes your commands, and then prints the results. It just evaluates it and the loop, it repeats the process. So this is actually a single line of code in this language called Lisp from the 50s. It was one of the first dynamic programming languages. And you could bootstrap an entire programming environment using a single line of code. You literally typed read-eval-print-loop, and we'll bootstrap a conversational interface. It’s between you and the computer, you're typing code. Now, composition interfaces are all the rage today. But ultimately, they've been around for a long time. I think DoS is a kind of REPL, a kind of composition interface. I remember just being enamored by the fact that you can sit in front of this machine, have an idea and type some symbols into a keyboard, and then have this machine, interpret what you want to do and do it for you.
It's just like a magical thing and I think we just don't appreciate it anymore. It's amazing we invented this thing. And my father, with a single finger, could sort of type these things and run into errors and look into manuals, you know, when you bought a computer in the 90s, he would get these manuals. And they're actually pretty cool because they taught you how to program and all of that. And so after he's done kind of exploring the computer, and he would like, be very frustrated because he didn't know how to use it, he would turn it off and go to bed. And then I would sneak back into the room. And I would turn the computer back on. And then repeat the commands that he did, except I would do it right. And so I learned how to use the computer by looking over my father's shoulder. And then one night he caught me, and he got so mad because he thought that I was going to destroy this machine that he poured his life-saving into. As you know, it meant a lot to a poor government worker, and he was visibly angry at me. But then I told him no, like, relax, I know how to use it. And then I started teaching my father how to use it. So I could write basic batch programs. So like the DOS or Windows programs at the time
Kyle:
How old were you?
Amjad:
I was six years old at the time. And so, you know, I immediately became sort of the computer kid in the neighborhood, and in our community. This really began my journey with computers.
The first time I did any programming in any professional capacity, was building a business when I was 15. Actually, I started programming when I was 13, but I was really, really into counter-strike games. I would spend all my time at these land gaming cafes, the internet in Jordan was like dialogue up until like, you know, 10 years ago. And so you'd have to go physically to a place and play games. And, you know, all these places, didn't use software to manage their computer stores or the games. It was manual that I would have to write down my name, and how many hours I paid for. I imagined that I could use a computer to do this. So I started writing software for that. And I started selling it to everyone at 15. And it made a lot of money, just building this business where I managed these land gaming cafes and wrote software for them and all of that.
The last pivotal moment in my upbringing that led to replit was when I was going to school in Jordan. I was going to university to study computer science, I didn't have a laptop, laptops were still fairly expensive at the time. And so that was like 2005 to 2008, something like that. And the first thing that I ran into, as I'm sure a lot of people still run into, is you'd have to install a program in the development environment. So you want to do like a Java course, you'd have to download a one-gigabyte Netbeans IDE. And then you'd have to configure that thing. And then God forbid you to share the program with someone they're missing a DLL or something like that. It was this huge pain in the ass. I was like, this can't be the case. There must be something that actually allows me to like write code online and allows me to wholesale online. I started searching and there was none of that in like 2008 and so I started naively sort of building an online REPL and got something working pretty quickly, the first version that I got was a text area, and you would write a bit of JavaScript, you would hit, click run, it will evaluate the JavaScript, it will alert it using the native window alerts method. And that was a REPL, right? And I started using it. And I gave it to my friends and you could save the program and share it. And it went viral at my university.
It was like, wow, like, I just wrote this, like in a day, and this is already something that's very useful and people use it. So I continued going down that path. I hit a roadblock at some point, which is, I wanted to add more languages than JavaScript because the browser only ran JavaScript. And at the time, we didn't know what to do. I started writing programming languages. I started by saying let me write this schema in JavaScript, you could get a Lisp or scheme JavaScript done pretty quickly. But then, I wanted to add Python, and that was really hard. So I tabled the project, and a year went by, then a year later. I found that Mozilla had started working on this project that allows you to run arbitrary code in the browser. Now it's called WASM, but at the time, it wasn't called WASM. It was just called Emscripten. And it became the first project to build something useful on top of Emscripten. So we compiled Python, Ruby, a bunch of languages, and JavaScript, and then released this open source project called Jess - Node.js Repl, and it was the first sort of multi-language and browser online Repl.
This was around 2010-2011. This project went viral, it was on Hacker News, and it was everywhere. And that got me my job in the United States, I came here as a founding engineer at Code Academy. And then I worked at Code Academy for a while, and then at Facebook before we started replit. And so the journey of replit has almost been inspired by that six-year-old experience that had a gem of an idea. But really, since 2008, I’ve had the vision you should be able to write a program in the browser, or really any device, like any computer, you should be able to program wherever. The idea behind replit is to create the most ubiquitous programming environment in the world. Today, we have a mobile app, we just crossed 200,000 downloads, and it's going viral in India, as we speak. We have thousands of daily active users and just crossed over 10,000s. We have people using it all over the world on any device, working in multiplayer in real-time sharing programs. It's been this huge journey since 2008 where we want everyone to be able to write programs anywhere.
Vision for Replit
Kyle:
13.35: Your story is very much interwoven with what Replit is and they are pretty interconnected. Like you very much built it for your own experience to be able to offer it to people who are trying to learn all these different things such as those that don't have the time or all the resources to be able to get everything set up. We talk about founder product fit, when you think about the vision down the road of what Replit is capable of becoming, how do you think about that vision, and how would you articulate it?
Amjad:
It’s always been the case that we wanted to build something that was really meaningful, and not just something that you sort of kick the tires with and leave. We've been in discussion and evolution mode where we continue to add power to the platform, week over week, day over day. The way we run the company is literally on a weekly basis where we have something called weekly wins. So every week, people need to have an achievement. You know, everyone talking about Elon is like, what did you get done this week? Oh, that's how we operate. Since the start, we want to know what is happening with the product and the company on a weekly basis. If you're not evolving, you're dying.
In the case of replit, if we're not adding power, we're losing customers, we're losing users. And, so the problem is that there's like so much to build. I think we're at this stage where we're finally at the last few pieces of the puzzle before we unlock a massive amount of value and growth in the world. Our vision has always been this place where you kind of write your first line of code, meet your friends, and maybe create a startup online place where you can earn your first dollar, we just announced bounties two or three weeks ago. And the idea behind bounties is that if you're a programmer who just learned programming at replit, you should not have to wait months and years and go to computer science school to earn your first dollar, you should be able to do it in the first month.
If a kid, you know picked up programming on the replit app today, you know, somewhere in India. They should be able to go to replit and earn their first $10, $50 or $100 immediately. There's no There's absolutely no reason why people learn how to code when they're like 15-16 years of age. And then have to wait 10 years before they because they earn money, right? I mean, that like makes it impenetrable to a lot of people. And so the idea behind replit is like, how do you actually make software, the internet, this like deeply programmable thing. And also remove all the barriers to entry, and make it immediately monetizable. So anyone in the world should be able to have an idea for a piece of software, be able to write this, write the code for it, or ask the AI to write the code for it or ask other people to write the code for it or work with a combination of API's and people to generate that idea and create a product and immediately be able to earn money from it. So ultimately, and this is counterintuitive because everyone thinks we're developer tools. However, ultimately, replit is like a tool for economic empowerment. It is about reducing the distance between ideas and wealth creation. That's really what it is.
It's that you have an idea for how, and then how long does it take you to create wealth in the world. And so we think in the future when you project out everything that's happening in AI, and you know, GPT-3, crypto and all that stuff. I wrote a thread about this the other day, but what's going to happen is that literally, the people who can think the most and can have the most well-formulated thought will be able to be rich. That's like really the future. The returns to cognitive ability are going to be huge, like anyone who is smart and relatively hardworking, will be super rich. And it will be not, it will not be encumbered by your race, religion, whatever it is that's previously been distributed around the world, it will not be focused on any kind of locality. Things like Starlink and other projects will power the world with the internet, everyone will have devices that could access the internet, and therefore accessing powerful computers like rapid access powerful AI or Ghostrider will be easy for people to just create their dreams. And we want to embed economic incentives into the ecosystem.
We just created this digital token called cycles. cycles is not a crypto, it's a centralized currency. You can buy products from replit using cycles. It's named after computer cycles. So like CPU works. The idea is that imagine in sci-fi movies when everything is in abundance where we have AI and they're sort of running the world, the thing that is actually most scarce is computer cycles. And so the replit currency, we think is going to be the most scarce currency in the world because it represents a computer cycle. And, and you could earn it today, you can go and do bounties and earn it today. But that's like not the end vision for this is really just like the start, we think in the next year or two, you'll be able to compose applications on replit through the combination of people and AI, and have monetization built into the system from the start. Today, people build applications and then hope to monetize them in the future.
And this is like how the whole venture industry is sort of built, like, you create some kind of value in the world. And then you figure out how to capture value by selling. And I think it's a broken market in a way. What I think should happen is that any intellectual property should be monetized from the start. So I'll give you an example. If you write a hyper-optimized sort function, it should be able to link to your program from my program, and I should be able to pay you per call. The programming goes from this world where it's stack-based, where I collect software from GitHub and the internet and other people. I put it in a stack, I ship it to AWS, and AWS runs it somewhere else. And then you know, Stripe monetizes it for me, that's not how it's going to be, it's going to be more like network-based programming, where I could link into any other software into the ecosystem, all the software is running all the time. And the currency and value are built into the working of the software.
On The Future of Education & Conformity in Society
Kyle:
22.09: One thing I wanted to double-click on is that you were talking about this vision of bringing the next billion programmers online, right? It's sort of this democratization of allowing everybody to be able to be software creators. So both as a parent, and as you think about the next billion folks who are your users and your customers and stuff, what do you think are the characteristics that we need to help endow the next generation with, if you will, to be able to help them more readily set through that door?
Amjad:
I will like to take the negative to that question, which is - what are the characteristics we need to back away from right? And I think extreme conformity has been the thing that made the industrial world work. The industrial world is predicated and designed based on factory assembly lines. It is based on conformity we needed as humanity, we needed humans to work in the same way that component parts of a manufacturing process work the same way. That's why you see everything in society as kind of structured, like an assembly line, right? So for example, school is literally an assembly line, first grade, second grade, third grade, fourth grade, go all the way up, you know, go to college, and you're being passed into this assembly line. People work on you a little bit and then at the end, you're like this perfect robot, and you go, you say the right things.
And you're literally an actor, like you go into corporate America, and you are expected to act and to say the right things, and live a good life. There's this book called Bullshit Jobs. Have you read it? So this writer estimates that 50% of jobs are bullshit. If you look at it, ask your friends. In tech, we actually have it a little better, but you know, most people think their jobs are useless. And the reason is that like its this extreme conformity and this idea that people do what they're expected to do. And I think what we need to do is like really move away from that almost completely. I think we need to really redo the education system. And, I think we make the world away from the image of the factory in the assembly line. If you want to teach something to your kids, teach whatever the opposite of conformity is, and I think maybe that's being more free-thinking, being more radical, being more weird. Being more adventurous. Not worried about what, you know, what other people think of them. Just, you know, to some extent, you're gonna go too crazy and like, create monsters. Although that's probably better than just like, complete, conformist.
So I mean, we need better systems. Did you invest in Synthesis?
Kyle:
Yeah, we are proud co-investors.
Amjad:
So in 2018. We got this invitation from SpaceX. Turns out Elon’s kids are using Replit to learn how to program and it's like, okay, well, yeah, we'll go to SpaceX no big deal. And so we went down to SpaceX, Haya, my co-founder and wife. And we hung out for an entire day, the school is literally what I was just talking about, which is the anti-conformity school. They did away with classes. Everything is self-driven by the kids. They learn by playing games, literally, they design games and play them, they have these things called simulations. During these simulations, you can solve these big problems in a group setting. It was obvious to me that this is the future of education. Like, I think as much Elon is in the news today and everything. I think this guy's actually somewhat underrated, which is a crazy thing to say, like, overrated is like, no, it's actually he's able to find novel solutions in totally different areas. I don't think he's gotten credits for what he did Ad Astra that school shut down. The head teacher, the co-founder of that school with Elon left and started a startup called Synthesis, it took the same program and put it online and you can put your kid in it, you're all young. But you know, if you have a seven-year-old and above kid, you can put it in there. And they'll learn problem-solving techniques, and learn how to work with other people. And I think that's like one real example of the future of education. But I think we need an entire stack outside of the traditional education system.
Open-Source AI
Kyle:
28.16: I want to talk a little bit more about your Ghostwriter platform, and what you guys have done there. And like, more broadly, what feels unique because we've had these breakthroughs in AI in many different ways, what feels different about this moment? I've heard a couple of folks talk on this, like Sam Altman and others, it feels like open source is keeping pace with closed source in a way that hasn't really happened before, where a lot of times a lot of the innovation is happening, either in academia, or is very theoretical, or it's happening in massive companies. And it can take sometimes years to trickle down to other folks. It feels like every time we make a breakthrough, it immediately is disseminated.
With Ghostwriter, you guys wrote a great blog talking about some of the models that you use to build an open source. And so I'm curious to talk about how did you guys approach that? And more broadly, what do you think about the role that open source is playing in the sort of Cambrian explosion of AI that we're seeing now?
Amjad:
I've done things with code all my career now. I mentioned I did write IDEs, compilers and interpreters at Yahoo. I also worked on libraries and things like that at Facebook. I also worked on what became the Metro Bundler, the main bus little behind, React Native, I worked on jest, I worked on Babel. And everything that I've done was things that manipulated code. And one thing you realize about compiler technology is that it is very tedious and classical, almost Algorithmic in style. So when you're writing compilers, you're dealing with trees, and you're manipulating trees. And you're just working in a very sort of classical algorithmic context. AI and ML actually haven't touched that industry at all until last year, which is wild.
But to me, it was always obvious, so I started writing minifier at Facebook, just before I left, and I spent all this time writing an optimizing compiler. And it just felt to me like you could totally do this in a machine learning context, you could totally learn algorithms just from the output. And you've started playing with NLP there. The idea is you can apply NLP to programming. And I read this paper called on the naturalness of software, which came out in 2012 and became sort of post facto, the seminal paper on how to use NLP in programming. But basically, the authors found that if you look at code, it really resembles natural language. We literally invented a programming language based on how we talk. And yes, it has structure and all of that, but it has a lot of repetition. It has a lot of idioms, it has a lot of patterns. And so when like, GPT-2 came out, I started playing with it. It became pretty obvious that this thing can code. It was clear that it was like, oh wow, this thing can learn enough language that it also learned enough structure of language and then we started playing with it. But it was really hard to get anything done with it.
Until we had GPT-3 come out, you could just give it a prompt and it just writes code for you. And that really was a time when you would just say I think 2020 the future of software changed forever. And basically, these language models, not only learn the structure of language, but they also learn things outside of their training distribution, meaning that they exhibit creativity and generality. And they have this thing that's called in-context learning. We call it prompting. And the idea is that you can change its behavior based on the context. And so basically, we suddenly have this alien intelligence, that is, that a machine can learn a language very well, that we can sort of semi-direct it in really interesting ways. It can write code really well. So that's the best way to think about LLMs. Think of them as alien intelligence.
When I work with it, I really try to have a bit of a theory of mind about it. Like how it's thinking. If you think about it, as a human, you will fail. If you think about it as a computer, you will also not do very well with it. Just think about it as a different kind of intelligence. And, you know, we built a few things on top of GPT-3, but one thing that became very obvious to us is that we needed something that's cheap and low latency. Yeah, because especially we were working with the mobile app. And we had this idea of bringing the next billion developers online. And when we are able to get people in, in places where, you know, they're on their phones, and the internet is not that great to be able to write code and get completions on their phones. So we sort of scoured the internet, and we found this small project from Salesforce, called Code Gen. This was the Salesforce research project that generated code that was similar to GPT-3, but just more focused on code. The problem was that it was really slow. For you to get like a full completion, you had to depend on the GPUs, but it was over 10 seconds for us. And so we had to literally get it two orders of magnitude faster and better.
At Replit, anytime we start a new project, we treat it like a startup. We go into a room, and with three or four people, go as hard as you can and make progress week over week. And so you know, the first week, we did a bunch of optimizations, we rewrote it in C++ and changed the architecture. And then the next week, we added streaming and batching. And then added these performance optimizations. Eventually, we got it down to like the first line of completion. Now, the median time is 350 milliseconds, which I think is the best in the industry right now.
We also figured out that the Salesforce model was undertrained. And we started training it, even more, we started collecting data and training it to become better. And all in all, it was like eight weeks between finding the project and deploying it in production before getting people to buy it. We now have over 1,000 customers within a couple of months of doing it. And it's growing really fast and it has become one of our fastest growing products. We have a new version coming within the next couple of days that is better trained. We’re also working on another version, that we're training our own model from scratch. We think that's going to be even 2x or 3x better than Ghoswriter. We also have a really big version coming next year. And I'll put a pin there, in case you're interested in talking more about that later.
The amount of creativity in the world is really unprecedented. You can go back to what I said earlier, about people being able to have ideas and execute them in real time, it's already happening. Replit will even make it faster. But today, when a paper drops, there's literally a 24-hour period until it gets implemented. When the DALL-E mini paper came out, literally within 24 hours later it was rapidly downloaded. The same happened with stable diffusion. The same as every machine learning paper that drops, there are people that are just waiting on GitHub to go and implement those.
There is this huge amount of pressure amongst creative people wanting to apply their creativity, and they do it for free. I think these people should be able to earn from it. It's a bug, not a feature.
One crucial thing there is that I do think that the big companies do have somewhat of a monopoly on really big models. I don't think we're going to find a GPT-4 open-source alternative from an upcoming company. I think there's going to be this bimodal distribution, where the small models are very hyper-specified and companies can train startups can train them from scratch. But on the other end, you're going to have the 1 trillion-plus parameter models. For better or worse, those are going to be like the OpenAI’s of the world. They've done a great job of ChatGPT. It's up to you if you haven't played with it yet. Just lock an entire day because it's really compelling.
Building a Culture of Accountability & Autonomy
Kyle:
38.46: Let’s talk about Replit’s culture of decentralization, accountability and freedom. What does that look like in practice to be able to give people that level of autonomy and freedom? Replit put out their accountability framework. What does it look like to give people a high level of freedom and accountability in practice?
Amjad:
This goes back to the evolution from the industrial society to the information society. The hierarchal system is probably even older than the industrial age. It's more like an agricultural society. We can think of the way humans are organized as a technology and apply this analogy to the hierarchy. Technology is not just gadgets, it's also intellectual. Hence, organizational tools are technologies. The hierarchy is a technical innovation of the Agricultural Age.
For the first time, it goes from this completely egalitarian, flat sort of hunter-gatherer system to the sort of farming society where you had to have separation of duties. You had to have reporting structures, you had to have different accountability. And you had to have a large swath of people that were totally replaceable. And you had to know kings, peasants and everything in between. The hierarchy was a very important tool. I think it got carried on into the Industrial Revolution as well. But it was never questioned. However, I do think that hierarchies like these will continue to be important. But open source shows that there are other ways of working. If we look at Linux, or Bitcoin. Bitcoin was at some point a trillion-dollar project. Linux is probably a multi-trillion-dollar project. Linux, if you just count all the servers of the world, it's probably almost a 5-10 trillion dollar technology. It was started by this grumpy dude from Finland, and then it blew up and worked with people all over the world in his bathroom. Linus is known for working from his bathroom and creating trillion dollars of value. It's amazing technology and creativity.
If you look at companies like GitLab, where they never had an office. They are completely asynchronous. They work all over the world. On the other hand, you look at companies like telegram which literally has a ship that they sail around the world. Did you know this? Yeah, it looks it up. Telegram was started by a dude in Russia who Putin wanted to kill. He started a Facebook clone called VK and it got too popular in Russia. So the government took it over. He ran away from Russia, and then built Telegram and operates the company from a ship. Now they're in Dubai. But like all of this to say is that there are now fundamental new ways of organizing companies. If you survey the landscape, you can see that some of the most successful companies are actually very weird. And Facebook was very weird at some point. I think they lost their ways and markets and trying to bring it back now.
I think that more of a loose structure rather than a self-organizing structure is probably a better fit than a strict hierarchy in this more information age where things are more malleable. My brother Ferris who works at Replit was trying to explain to someone how decisions get made at Replit. He said it gets made by memes. He meant that we have discussions, and then memes start to emerge or sort of ideas that start to percolate in the organization. And suddenly they become real things that we have to go execute on. They sort of become apparent over time. Now, every now and then I have to sort of dictate things and I have to do top-down things. But actually, it's fairly rare that I have to do that. And we generally make the right decisions. I just like to give a speech every Friday and things happen that are somewhat correct. And so it's really hard to describe it just like a very collaborative and open culture. Every project has a single DRI (directly responsible individual). Every project has sort of a monarch that is responsible for all the decisions. The thing you have to worry about is committees. Humans love to create committees for whatever reason. When you see a committee, you have to kill it immediately. Because like a snake, you know, it's an apple, it starts with one employee and continues to grow before suddenly, it becomes a big organization. But what you want is to have autonomous people with a network around them to help them execute and work.
Audience Q&A
Participant Q&A #1: Why Build Ghostwriter The Way You Do?
My question is about the Replit Ghostwriter. So I get that you can train your own models and Code-Gen stuff. But like, why not place your bets on GPT? Is this because of margins? Is it about latency? Is it about monopoly? Like what's the reasoning behind it?
Amjad:
We have a disease at Replit called non-invented here and that's the honest answer. I can sort of rationalize it. But latencies is a big one. Right now, we have about 5,000 weekly active users on Ghostwriter. It is all run using only three GPUs. At Replit, our core competency is running cheap compute at scale. It just didn't make sense for us to offload that to another organization, we do use OpenAI on the back end, for some other stuff. These include some of the features that you could wait for. So when you click Generate, and you wait 30 seconds, that goes back to GPT-3. However, for the interactive features, we had to do them in-house because we wanted to do work to optimize all performance. We wanted to have total control. Like at some point, we're going to be able to train models that we can put on phones, and we just have to build that capability.
The other thing I will say about transformer technology is the transformer model and the underlying model behind GPT-3. All these models are fundamental innovations in software, and I think every company needs to build some kind of expertise in it. We're using Bert, for example, in search. We're using Bert and Spam. So I think, you want to use some of these larger models. But if you don't have some capacity to deploy and run these models, I think companies will lose if they don't develop that internally.
Participant Q&A #2: The Future of Software, AI and Virtual Assistants
Right now the status quo is we're kind of in a situation where AI is assisting coding so specialists, coders, and software engineers can actually use the likes of co-pilot, as well as ghost writer, do you think we will eventually reach a future where AI can help abstract this away from code where code can be dealt with back? And I think we've already seen some applications like thought spot in the context of business intelligence, obviously, you can spit up AI and ML models using no code, but what do you think the future looks like in terms of abstracting that away?
Amjad:
I think everyone will be able to generate software by the end of this decade. It's a rising tide that will lift all boats. I think the best programmers will become super exceptional programmers. But everyone will get better. The layperson will get better. I made this analogy on Twitter which got a lot of pushback: “Everyone in the world will be at least John Carmack-level software capable.” What I meant was that you'll be able to talk to Siri and have Siri generates software for you on the fly. You can use something like Siri or ChatGPT which can make an app for you. Almost like having these models build mobile apps for you so that you can use them immediately. It can crawl the web and do things on your behalf.
I will just make a prediction that I think before the decade runs out, you'll be able to talk to some kind of chatbot and give it a very complex software task. For example, pull all my connections from LinkedIn, put them in Gmail, generate an email, pull this image from there, put that image in the email and say x, y and z. You're able to just say what should happen. And on the back end, it's not going to be like Siri, pre-programmed tasks like the way it works today with shortcuts. Instead, we’ll have these “If and Then” statements. What's going to be happening is that Siri-like assistants will actually be very competent coders. When you make requests of them, they'll be able to go generate that code and execute it for you in a rapid-like environment. In general, I think that's one thing, I think you'll be able to get to run things like cron jobs, as well, and a lot of things that programmers take for granted that the average people can do, like, I can't ask Siri to like do a cron job today. I believe that before the end of the decade, your mom will be able to ask Siri to do something on the loop and do it every other day or whatever. I just think its going to be a great world.
Participant Q&A #3: How do startups manage bets and failure?
As a YC and Paul Graham student, how do you validate that what you're working on is really going to add value to your customers, through your strategic planning? How do you make that connection?
Amjad:
There are things that are fundamentally bets. For example, bounties could end up being a flop. But the way we're approaching it is like, can we grow it? At least 10%? week over week, right now? It's actually more than doubling week over week and the usage is really ‘high-value usage.’
And then you have to be honest with yourself. If it's not working, is it because of a bug in the product? Or is it just a fundamental lack of product market fit? And I think that's the real art. It's really difficult to do actually, with Ghostwriter, for example. It was like an immediate product market fit, we still have to look at the churn and see how good the product is. But if we can see a world where we can continue to make the product better and better. And it's obvious to us, we can make that bet as a company. However, there are things that are fundamentally still bets and we'll see how it works out.
There are times when you are so steadfast and your vision that you want to do it anyways. For example, we’ve been told that nobody wants to code in the browser, and we've been told that it is technically impossible. We've also been told all these things over and over again. And you just trust your gut and do it. So I would say it really depends on the context. But ultimately, it's about rapid iteration, talking to customers, and staying close to your customers. And like building something for yourself. If you're gonna like it, then the likelihood of other people liking it is pretty high.
Participant Q&A #4: How does Replit structure their teams?
Replit has this pretty great hacker culture which I think is quite different from the ML research community, which is not hacker-like in my experience, have you found some insights from those two as you spin up more and more capabilities?
Amjad:
Yeah, actually, what we found is that it's been pretty hard to hire talent from established ML companies. Our talent actually skews very young folks. The other thing that we found is that the LLM space is so different than traditional ML space. The skillset is very different. I mean, it's not too different, but it's somewhat different. But the mindset is really different. The way you're thinking about training is different than the way you think about the data. It's also so different from the way you deploy these products. So, we're finding that like new grads are actually a much better fit for us than established players. There are people that had been working in transformer LLM style for a while before this boom, those people would be a fit, but they are very expensive. They are the most expensive in the world. In fact, the people behind the transformer papers are all building companies now and you see them raise $100 million dollar seed rounds. In any case, we've been lucky that we have an advisor, who used to work at Google. He was one of the early people on LLMs for code. But other than that, we're doing what we're doing best which is to discover talent.
Participant Q&A #5: What are you focused on learning?
It was funny that one of the most common questions I got from every one of your investors except one that everybody talked about how you're a pretty voracious learner and constantly thinking about what's the new thing you're trying to tackle. And so everyone's gonna ask, what are you learning right now? Like, what's the challenge that you're tackling? Or the thing that you're trying to better understand?
Amjad:
I've been so obsessed with this AI stuff, that it's probably still that. I'm actually trying to understand the fundamentals of how transformers and how large language models work. It just feels like a mystery as to how these things work. If you don't know how language models and transformer base language models are trained. It is basically that you give them a large corpus of data. And then you mask one before you then you ask them to guess that word. And if their guess is wrong, you sort of back-propagate and you change the weights of the neural network. That's it. And you get all this magic, by just doing that for a really long time for a neural network with a lot of parameters. It results in emergent behavior to me that is a real mystery as to how that happens.
And in that mystery, there are probably answers about how our brains work, and how we learn. What is what is consciousness? These are really fundamental questions about what is intelligence and it is very interesting to me. It's something that I've been obsessed with all my life since I was a kid. I’ve zoomed in on these fundamental questions. Like, this is like a weird thing, like the fact that we're experiencing the world. Just even though my experience is different from your experience, it's all a mystery to me. We don't know how to explain it and we don't know how to study it. I’ve always been interested in sort of reading about that. And then obviously, the other big questions, which is like, where did the universe come from? Why is there something rather than nothing? And, but yeah, I think the work that's happening in intelligence and AI is so fascinating and has been really taking up a lot of my time.
There are also things I do like continuously trying to like learn basic things on how to do my job better. I like to read strategy books and business books. One of my favorite books is called “Seven Powers.” If you look at Replit, we are trying we're trying to build a lot of moats in the business, perhaps no faults of ours. As you know, I think in Silicon Valley, business strategy books are sort of a little underrated because everyone had this idea that execution is everything that matters. So I typically have in my stack, some kind of business strategy book that I that I'm going through.
Kyle:
Amjad, this has been super fun. Thank you very much for your time.