Chris Martin

  • Open source large language models

    Stanislas Polu has some predictions about the state of large language models next year. The only prediction of his that I feel especially confident in myself is the following:

    There will be an opensource Chinchilla-style LLM released this year at the level of text-davinci-*. Maybe not from the ones we expect🤔This will obliterate ChatGPT usage and enable various types of fine-tuning / soft-prompting and cost/speed improvements.

    I would say that now, especially after the success of ChatGPT, an equivalent open source LLM will almost certainly be released in the next year. This will likely follow the same pattern as image generation AIs earlier this year: first, OpenAI released DALL-E 2 as a private beta. Then, a little while later, Stable Diffusion was released which, although it wasn’t quite as good as DALL-E, was free, open-source, and widely accessible. This allowed for an explosion of creative applications including photoshop plugins, 3D modeling plugins, and easy to install native frontend interfaces.

    While I believe text generation AIs will have a similar moment early next year. The unfortunate truth is that running even a pre-trained text generation network requires significantly more computer memory than is required to run similarly sized image generation networks. This means we will probably not see early open-source text generation networks running natively on consumer hardware such as iPhones like we have with Stable Diffusion (although it is possible Apple will, once again, help with that).

    My hope is that these developments in text-generation spur some much-needed innovation in household voice assistants which are increasingly feeling dated.

  • AI collaborators

    Onformative Studio shares its expirence using a generative AI to create 3D sculptures:

    We were guided by the question of how our role as the designer is changing when human creativity and artificial intelligence co-create. In the course of the AI’s sculpting process we were inspired by the unpredictable strategies and outcomes of the reinforcement learning: an experimental approach par excellence, which we guided, observed and visualized.

    Above all, we have also questioned our own role as creators. Rather than leaving creation to AI, we need to find ways to integrate it into the creative process. We take technology as a starting point, a tool, a source of inspiration and a creative partner. The human aspect is quite clear in this: We choose the rules and define the approximate output. However, in the end it was the interplay between our human choices and the agent’s ability to find the best solutions and occasionally surprise us. This made the process rewarding to us and shows the true potential of an AI based co-creation process.

    This reminds me of some of the ideas Noah Smith and Roon laid out in their recent article about generative AI:

    We think that the work that generative AI does will basically be “autocomplete for everything”.

    What’s common to all of these visions is something we call the “sandwich” workflow. This is a three-step process. First, a human has a creative impulse, and gives the AI a prompt. The AI then generates a menu of options. The human then chooses an option, edits it, and adds any touches they like.

    There’s a natural worry that prompting and editing are inherently less creative and fun than generating ideas yourself, and that this will make jobs more rote and mechanical.

    Ultimately, though, we predict that lots of people will just change the way they think about individual creativity. Just as some modern sculptors use machine tools, and some modern artists use 3d rendering software, we think that some of the creators of the future will learn to see generative AI as just another tool – something that enhances creativity by freeing up human beings to think about different aspects of the creation.

    I am optimistic that generative AI will continue to make creative expression of all kinds more accessible. At first, AI assisted design will be viewed as lower status or less authentic, like baking brownies using a pre-made box mix instead of “from scratch.” Later, though, once these technologies become more established, not using an AI will prompt discussion and warrant explanation — AI collaboration will be the norm. This is already the case with smartphone photography.

  • Technology and extrinsic motivation

    Matt Yglesias on how technological progress might change higher education:

    The past couple of centuries have seen a steady increase in the market demand for certain kinds of skills. That’s meant people want to acquire these skills even if they don’t necessarily have tons of inherent motivation — they want training to earn a better living, and education as a formal process is a very useful motivational crutch. It’s at least possible that the next twist in IT will be to rapidly erode the demand for that kind of education, meaning people will be left primarily to learn things they are curious about where there is much less need for external motivation.

    This is part of the reason why I believe fostering a strong and diverse set of intrinsic motivations is crucial at all levels of education. Whether or not the future Yglesias describes here comes to pass, students driven by intrinsic motivations are always going to be more passionate and engaged than students driven by extrinsic motivations.

  • The professionalization of GPT

    Sam Kriss on how, as GPT models have become more accurate, they have become less compelling:

    I tried using GPT-2 to write a novel. I let the AI choose the title, and it was absolutely insistent that it should be called “BONKERS FROM MY SLEEVE”, with the caps and quotation marks very much included. There was a Pynchonian array of bizzarely named characters (including the Birthday Skeletal Oddity, Thomas the Fishfaller, the Hideous Mien of Lesbian Jean, a ‘houndspicious’ Labradoodle named Bam Bam, and a neo-Nazi cult called ‘The Royal House of the Sun’), none of the episodes made any sense whatsoever, and major characters had a habit of abruptly dying and then popping up again with a different gender. But along the way it produced some incredible passages

    …

    GPT-2 was not intended as a machine for being silly, but that’s what it was—precisely because it wasn’t actually very good at generating ordinary text. The vast potential of all possible forms kept on seeping in around the edges, which is how it could generate such strange and beautiful strings of text. But GPT-3 and its applications have managed to close the lid on chaos.

    …

    There are plenty of funny ChatGPT screenshots floating around. But they’re funny because a human being has given the machine a funny prompt, and not because the machine has actually done anything particularly inventive. The fun and the comedy comes from the totally straight face with which the machine gives you the history of the Toothpaste Trojan War. But if you gave ChatGPT the freedom to plan out a novel, it would be a boring, formulaic novel, with tedious characters called Tim and Bob, a tight conventional plot, and a nice moral lesson at the end. GPT-4 is set to be released next year. A prediction: The more technologically advanced an AI becomes, the less likely it is to produce anything of artistic worth.

    I could also see this as something that would help establish the validity of AI assisted artwork. As in “AI could never make something this inventive on its own, the artist must have played a large role in the artwork’s creation.”

  • 2022 W49

    Next week is the last week before winter break, not that I am counting.

    I made some homemade pizza in the outdoor pizza oven on Saturday. For the first time my gluten-free dough tasted the same as my glutenous dough. I hope this means my gluten-free skills are improving and not that I simply don’t know how to cook with gluten anymore. Anyway, the pizza was good and I don’t think I will have many more opprotunites to use the pizza oven until spring.

  • What would a forward-looking technology education look like?

    I teach a technology class to students from Kindergarten to fifth grade. The accelerated development of truly impressive AI models recently — especially ChatGPT and Stable Diffusion — has made it clear to me how dramatically different technological literacy will be when my students eventually enter the world as adults.

    As we move into a future with increased technological automation, forward-looking curricula across all subject matters must focus on fostering creativity in students. Although AI can make new connections between elements in its training data, it is humans alone that are capable of generating truly novel ideas.

    I believe teaching young students how to code will continue to be important to some extent. However, with the rise of code generation technologies like GitHub CoPilot, the most durable programming skills might be spec-writing, debugging, and revising. Physical electronics and robotics will arguably rise in relevance for the foreseeable future. Understanding and debugging systems will be an important skill here, too.

    It would be great to hear from other educators that are thinking through similar topics right now.

  • AI and the future of education

    Zero Trust Homework

    Here’s an example of what homework might look like under this new paradigm. Imagine that a school acquires an AI software suite that students are expected to use for their answers about Hobbes or anything else; every answer that is generated is recorded so that teachers can instantly ascertain that students didn’t use a different system. Moreover, instead of futilely demanding that students write essays themselves, teachers insist on AI. Here’s the thing, though: the system will frequently give the wrong answers (and not just on accident — wrong answers will be often pushed out on purpose); the real skill in the homework assignment will be in verifying the answers the system churns out — learning how to be a verifier and an editor, instead of a regurgitator.

    I am not sure I fully agree with Ben’s proposal here but, at the same time, I am having trouble coming up with any coherent solutions for homework / assessments that truly account for the AI we have today — let alone what we will have 10 years from now.

    Ultimately, I am hopeful these advances in AI will push us to re-evaluate our current approach to education and lead us towards more meaningful, interdisciplinary, and practical approaches in the future.

subscribe via RSS