stable diffusion

Who’s in the loop? How humans create AI that then creates itself

If you think about the approximate lifecycle of AI that’s being built today, it goes something like this:

1. Write algorithms (eg, neural nets)
2. Scrape data (eg, text and images)
3. Train (1) algorithms on (2) scraped data to create models (eg, GPT-4, Stable Diffusion)
4. Use human feedback (eg, RLHF) to fine tune (3) models – including addition of explicit rules / handicaps to prevent abuse
5. Build products using those (4) fine tuned models – both end-user products (like MidJourney) and API endpoints (like OpenAI’s API)
6. Let users do things with the (5) products (eg, write essays, suggest code, translate languages). Inputs > Outputs
7. Users and AI owners then evaluate the (6) results against objectives like profitability, usefulness, controllability, etc. Based on these evaluations, steps (1) through (6) are further refined and rebuilt and improved

Each of those steps initially involved humans. Many humans doing many things. Humans wrote the math and code that went into the machine learning algorithms. Humans wrote the scripts and tools that scraped the data. Etc.

And very steadily, very incrementally, very interestingly, humans are automating and removing themselves from each of those steps.

AI agents are one example of this. Self-directed AI agents can take roughly defined goals and execute multi-step action plans, removing humans from steps (6) and (7).

Data scraping is mostly automated (2). And I think AI and automation can already do much of the cleaning and labeling (eg, ETL), in ways that are better cheaper faster than humans.

AI is being taught how to write and train its own algorithms (steps 1 and 3).

I’m not sure about state of AI art for steps (4) and (5). Step 4 (human feedback) seems hardest because, well, ipso facto. But there are early signs “human feedback” is not all that unique, whether using AI to generate synthetic data, or to perform tasks by “acting like humans” (eg, acting like a therapist), or labeling images, etc.

Step (5) is definitely within reach, given all the viral Twitter threads we’ve seen where AI can build websites and apps and debug code.

So eventually we’ll have AI that can do most if not all of steps 1-7. AI that can write itself, train itself, go out and do stuff in the world, evaluate how well it’s done, and then improve on all of the above. All at digital speed, scale, and with incremental costs falling to zero.

Truly something to behold. And in that world, where will humans be most useful, if anywhere?

Just a fascinating thought experiment, is all. 🧐🧐🧐

These times are only gettin’ weirder.

Podcast notes – Emad Mostaque (Stability AI and Stable Diffusion) – Elad Gil: Short form videos coming “within 2 years at high resolution quality”; “Run Stable Diffusion on your iPhone by next year”

(started notes around 20min in)

Bad guys have AI – they’ll create deep fakes
Community can come together to have counter measures

Elad: similar arguments to regulate cryptography in 90s

4chan has been “red teaming” trying to get the worst out of Stable Diffusion – and it’s not that bad

Especially for LLMs, should have more diverse data sets, have inter-governmental agency to monitor it

Have authenticity tool to verify source of every generated AI output

Generative AI – what are some use cases that should exist
Ali v Tyson live, Lebron v Michael Jordan
Emad wants to remake final season of Game of Thrones

Anyone can create their own models – any person, company, or culture

You need better data, more structured data
Extend models to run on edge – eg, anyone’s computers, iPhones
Make small customized models
“Run Stable Diffusion on your iPhone by next year”

Create national models and communities around them – let them leap frog ahead

Lots of emerging markets went from nothing to mobile phones, now can go to AI models on the edge

How far from short-form videos?
Phenaki, Google — getting close
Chaining these models together – they’re like parts of the brain
“Within 2 years at high resolution quality”

$100B into this sector in next 5 years

AI before today was qualified data science
Now it’s a new type of AI – not AGI yet, but incredibly small and powerful
By the time his daughter’s in university, doesn’t need to write essays

He aims (for Stable Diffusion) to be a layer 1 standardized infrastructure – create Schelling point
Mission is to “activate humanity’s potential”
Take it to India, Indonesia – give it to very smart young people to make their countries better

When AGI comes, I hope it thanks him for buying so many GPUs to help bring it into being

Many of Google’s “jobs to be done” will be displaced

Crypto is interesting – he’s in it since 2012 – focused on decentralized identity, zero knowledge proofs
“Nature of crypto is literally identity”
In a world of infinite content (AI), crypto identity is important
Need to be careful designing crypto economic systems

A year ago, if he said what they planned to do with SD, people would say he’s crazy
Surprised by how far they’ve come, the ability of others to contribute
The activation energy has been the most surprising – “they’re just excited”

“Probably see biggest breakthrough from a 16 year old in Uzbekistan” – the global open access nature of it

Will completely disrupt social networks – will move intelligence from the core to the edge
Apple is doing this – moving to AI – moving to edge

Opportunity to have personalized AIs that work for us and with us

SD is applying for B corp status – mission based
Plan to spin SD into different Foundations

Did investment round in August – didn’t give up any independence – did with investors that are open sourced and mission aligned

Which industries disrupted first?
-Call centers
-Powerpoint, forms of visual communication
-Artist won’t be that disrupted – will enable new forms of art

This tech is amazingly powerful

After releasing Stable Diffusion – people encoded it in Japanese – lots of use cases like this

So far governments have been very friendly

AI powered teaching – like Neal Stephenson’s Young Lady’s Primer

Moving forward, only release safe for work models

Licensing discussions should be more open

Will have models across all sorts of languages – recently released Korean model

Galerie.ai – make generative art with multiple models

A little something we’re working on, you can try it here: http://galerie.ai/

It does two things

1. Returns results from a growing library of the best AI art

2. Lets you create AI art from multiple generative models (including Stable Diffusion 1.5, 2.0, 2.1, and MidJourney)

Let me know what you think!

We auto-suggest some of the trending prompts and popular art results on the home page, try it out!

For example here is ” photorealistic portrait, a young beautiful woman Goddess wearing Echo of Souls Skull Mask armor, skeletal armor bones made from 24k gold and silver metal intricate scroll-work engraving details on armor plating, skeletal armor, gemstones, opals, halo, aura, intricate details, symmetrical,”:

The Venn of blockchain and AI

I’ve been thinking about the relationship between blockchains and AI lately. Both are emerging foundational technologies and I think it’s no accident they are both coming of age at the same time.

Multiple writers have already expressed this view:

AIs can be used to generate “deep fakes” while cryptographic techniques can be used to reliably authenticate things against such fakery. Flipping it around, crypto is a target-rich environment for scammers and hackers, and machine learning can be used to audit crypto code for vulnerabilities. I am convinced there is something deeper going on here. This reeks of real yin-yangery that extends to the roots of computing somehow

From Venkatesh Rao: https://studio.ribbonfarm.com/p/the-dawn-of-mediocre-computing

I think AI and Web3 are two sides of the same coin. As machines increasingly do the work that humans used to do, we will need tools to manage our identity and our humanity. Web3 is producing those tools and some of us are already using them to write, tweet/cast, make and collect art, and do a host of other things that machines can also do. Web3 will be the human place to do these things when machines start corrupting the traditional places we do/did these things.

From Fred Wilson: https://avc.com/2022/12/sign-everything/

In both writers’ examples, blockchain helps solve some of the problems that AI creates, and vice-versa. I’m reminded of Kevin Kelly who said, Each new technology creates more problems than it solves.

Blockchains and AI have a sort of weird and emergent technological symbiosis and I’m here for it.

So the brain flatulence below is just my way to think aloud, using the writing process to work through the question(s).

*Note: when I say “blockchain”, I include what Fred Wilson calls web3 and Venkatesh calls crypto; there are just a few canonical applications that we’re all familiar with (namely, bitcoin and ethereum); and when I say “AI”, I am thinking about the most popular machine learning models like GPT3 and Stable Diffusion

*Note also: I am just a humble user of these new and powerful AI tools, and can barely understand the abstract of a typical machine learning research paper; so part of the reason why I’m writing this is to find out where I’m wrong a la Cunningham’s law

—

A blockchain is a tool for individual sovereignty; while an AI is a tool for individual creativity

A blockchain operates at maximum transparency; while an AI operates largely as a black box

A blockchain clearly shows the chain of ownership and history; while an AI… (does something like the opposite in the way it aggregates and melds and mutates as much data as possible?)

A blockchain is “trustless”, in the sense that what you see on-chain is the agreed upon “truth” of all its users; while an AI is (?), in the sense that what it generates is more or less unique to the specific prompt / question / user (and even this can change as the model is updated, or new data is added]

An AI is much easier to use than a blockchain

An AI can create vast quantities of content, very cheaply; while a (truly “decentralized”) blockchain is limited by scalability and cost

An AI is centralized (to a specific company, or model, or data set) in the sense that decision making rests with a team or company; while a blockchain is decentralized and decision making is distributed

A surprising user experience – as in, an unexpected but delightful output – is typically net positive for a user of AI, while seeing something happen on a blockchain that you don’t expect would generally be pretty bad (yes, of course there are airdrops)

Blockchains are a competitive threat to industries with a high degree of centralization (such as fiat currency issuance, and payment networks); AI is a competitive threat to many individual online workers (such as language translators, and freelance writers, and basic QA/QC employees)

Both blockchains and AI have multiple open source products that can be forked by developers

Both blockchains and AI are platforms upon which many other products and services can be built

Both blockchains and AI are technologies that exploded into the popular consciousness in the last 10 years

Both “blockchain” and “AI” are very broad suitcase words, in part because they are both the product of many technologies combined in innovative ways: for blockchain that is everything from cryptography to smart contract programming to PoW mining to distributed consensus mechanisms; for AI that’s, uh…well everything listed here and more, I suppose

—

I’ll end here for now, but let me know what I got wrong, what I’m missing, and what questions or ideas this might inspire

—

Addendum #1: I asked ChatGPT

—

Addendum #2: This NYT article notes that SBF (“Sam Bankscam Fraud”) donated at least $500M to organizations researching AI alignment and AI safety. Not exactly the kind of symbiosis I want to explore, but worth noting.

—

Addendum #3:

Blockchains can only give precise answers, while AI can give approximate answers or even fabricate answers

Blockchains are censorship resistant, while AI is centralized (most are created by small doxxed teams) and have implemented restrictions on usage (most have rules against, for example, CSAM or nudity)

Interesting snippets from State of AI Report 2022

Full report here: https://www.stateof.ai/

I’m far from an AI expert, just an interested student who gets the tingly feels every time I use Stable Diffusion or see output from ChatGPT.

Snippets (copied verbatim):

–

The chasm between academia and industry in large scale AI work is potentially beyond repair: almost 0% of work is done in academia.

–

Finding faster matrix multiplication algorithms, a seemingly simple and well-studied problem, has been stale for decades. DeepMind’s approach not only helps speed up research in the field, but also boosts matrix multiplication based technology, that is AI, imaging, and essentially everything happening on our phones.

–

The authors argue that the ensuing reproducibility failures in ML-based science are systemic: they study 20 reviews across 17 science fields examining errors in ML-based science and find that data leakage errors happened in every one of the 329 papers the reviews span

–

many LLM capabilities emerge unpredictably when models reach a critical size. These acquired capabilities are exciting, but the emergence phenomenon makes evaluating model safety more difficult.

–

Alternatively, deploying LLMs on real-world tasks at larger scales is more uncertain as unsafe and undesirable abilities can emerge. Alongside the brittle nature of ML models, this is another feature practitioners will need to account for.

–

Landmark models from OpenAI and DeepMind have been implemented/cloned/improved by the open source community much faster than we’d have expected.

–

Compared to US AI research, Chinese papers focus more on surveillance related-tasks. These include autonomy, object detection, tracking, scene understanding, action and speaker recognition.

–

NVIDIA’s chips are the most popular in AI research papers…and by a massive margin

–

“We think the most benefits will go to whoever has the biggest computer” – Greg Brockman, OpenAI CTO

–

As such, the AI could reliably remove 36.4% of normal chest X-rays from a primary health care population data set with a minimal number of false negatives, leading to effectively no compromise on patient safety and a potential significant reduction of workload.

–

The US leads by the number of AI unicorns, followed by China & the UK; The US has created 292 AI unicorns, with the combined enterprise value of $4.6T.

–

The compute requirements for large-scale AI experiments has increased >300,000x in the last decade. Over the same period, the % of these projects run by academics has plummeted from ～60% to almost 0%. If the AI community is to continue scaling models, this chasm of “have” and “have nots” creates significant challenges for AI safety, pursuing diverse ideas, talent concentration, and more.

–

Decentralized research projects are gaining members, funding and momentum. They are succeeding at ambitious large-scale model and data projects that were previously thought to be only possible in large centralised technology companies – most visibly demonstrated by the public release of Stable Diffusion.