Tag: google

Truly valuable technology trends toward free and ubiquitous (another Kevin Kelly read)

This one’s also going in the personal bible archives…

Original source: https://kk.org/thetechnium/technology-want/

Some excerpts:

—

“there has been a downward trend in real commodity prices of about 1 percent per year over the last 140 years.” For a century and half prices have been headed toward zero.

—

GPS was a novelty luxury only a few years ago. It was expensive. As its technical standards spread into mapping services and hand helds, it becomes essential, and the basic service (where am I?) will become a commodity and free. But as it drops toward the free, hundreds of additional advance GPS functions will be added to the fixed function so that more people will pay ever more for location services than anyone pays now. Where-am-I information will be free and ubiquitous, but new services will be expensive at first.

—

As crackpot as it sounds, in the distant future nearly everything we make will (at least for a short while) be given away free—refrigerators, skis, laser projectors, clothes, you name it. This will only make sense when these items are pumped full of chips and network nodes, and thus capable of delivering network value.

—

Automobiles, like air travel, are headed in direction where all software and digital devices are headed: toward the free. Imagine, I said, if you could give away a very basic no-frills car for free

—

A car will move you from A to B, but it also offers privacy, immediacy of travel, a portable office, an entertainment center, status, and design joy

—

Google has the same opportunities with them that all producers have. They offer free commodities and charge for premium services. Search is free; yet they charge enterprises for custom Google search. Or they shift their customer from reader to advertiser; in Google’s eyes the chief audience for search is advertising companies, whom they charge

—

Technology wants to be free, as in free beer, because as it become free it also increases freedom. The inherent talents, capabilities and benefits of a technology cannot be released until it is almost free. The drive toward the free unleashes the constraints on each species in the technium, allowing it to interact with as many other species of technology as is possible, engendering new hybrids and deeper ecologies of tools, and permitting human users more choices and freedoms of use

Podcast notes – Noam Shazeer (Character AI, Attention is all you need) on Good Times w Aarthi and Sriram

Intro
-Founded Character AI
-One of authors of “Attention is all you need”
-Was at Google for 20+ years (took a few years break)

Went to Duke undergrad on math scholarship

Realized he didn’t enjoy math, preferred programming and getting computers to do things

During Google interview, Paul Buchheit asked him how to do a good spell corrector, and Noam ended up writing the spell corrector feature for Gmail

Google has been traditionally a bottoms up company – could work on what he wanted

When he started AI, exciting thing was Bayesian networks

Came back to Google to work with Jeff Dean and Google Brain team
“Just a matter of the hardware”
All the growth in hardware is parallelism

Neural networks are mostly matrix multiplications – operations that can be done well on modern hardware

Gamers / video games pulled GPU advancement (highly parallel hardware) out of market

Idea of neural networks has been around since 1970s – loosely modeled on our impression of the brain

Very complicated formula to go from input > output
Formula is made of parameters, and keep tweaking parameters
Neural nets rebranded as “deep learning”
Took off because of parallel computation and gamers

Neural language models are neural networks applied to text
Input is text to this point, output is prediction of what text comes next (probability distribution)
Infinite amount of free training data (text content)
“AI complete problem”
“Really complicated what’s going on in there” (in the neural network)

It’s a really talented improvisational actor – “Robin Williams in a box”

Model improvement is kinda like a child learning – as training and model size grow

Lot more an art than a science – can’t predict very well – if 10% of his changes are improvements, considered “brilliant research” – kinda like alchemy in early days

(Software) bugs – hard to know if you introduce a bug – the system just gets dumber – makes de-bugging extremely difficult

Co-authored “Attention is all you need”
-Previous state of art in LLM is recurrent neural networks (RNN) – hidden state, each new word updates the hidden state, but it’s sequential – slow and costly
–Transformer figures out how to process the entire sequence in parallel – massively more performant
-The entire document / batch becomes the sequence
-Lets you do parallelism during training time
–During inference time it’s still sequential
–

Image processing models – parallelism across pixels – convolutional neural nets (CNN)

Google Translate was inspiration – biggest success of machine learning at the time
Translating languages > one RNN for understanding, and another RNN for generating, and need to connect them
Attention layer – take source sentence (language A), turn into key-value associative memory, like a soft lookup into an index
“Attention” is building a memory, a lookup table that you’re using

DALL-E, Stable Diffusion, GPT3, they’re all built on this Google research

Bigger you make the model, more you train it, the smarter it gets – “ok, let’s just push this thing further”

Eventually need super computer
Google built TPU pods – super computer built out of custom ASICS for deep learning

Now need massively valuable applications

Turing Test, Star Trek, lot of AI inspiration is dialogue

Google LAMDA tech & team – eventually decided to leave and build as a startup

“The best apps are things we have not thought of”

If you ask people with first computers “what is this thing good for”, would get completely wrong answers

Parasocial relationships – feel connection with celebrity or character – one way connection – with AI you can make it two ways

Aarthi: “Your own personal Jarvis”

Still need to make it cheaper – or make the chips faster

Aarthi: ideas / areas for entrepreneurs
-Image gen has exploded – lots of good companies coming, very early and promising
-Things like Github Co-Pilot
-new Airtable – using AI for computation
–

Sriram:
-What’s optimization function that all these models will work toward?
-Will be a very big political / social debate

How do you know better than the user what the user wants?

Data aging or how technology amplifies our lived experience

Fascinating concept from VGR: https://studio.ribbonfarm.com/p/superhistory-not-superintelligence

To quote:

Those of us who have been using Google search for 22 years are /already/ like 100 years older than our biological age. Every year lived with Google at your fingertips is like 5 lived within the limits of paper books. In many ways, I feel older than my father, who is 83. I know the world in much richer, machine-augmented ways than he does, even though I don’t yet have a prosthetic device attached to my skull. I am not smarter than him. I’ve just data-aged more than him.

And:

But when Magnus Carlsen defeated Vishwanathan Anand in 2013, something weird and new was on display. I’m not a chess player, but from the commentary I read, it seems like not only was Carlsen more of a raw talent than Anand (same as Kasparov vs. Karpov) but he was also “older” in a weird way, despite being nominally 21 years younger. Carlsen is young enough to have been effectively “raised by AIs” — the most sophisticated chess AIs available on personal computers when he was growing up in the aughts. His playing style was described as kinda machine-inspired, pushing hard all the way through the end, exploring unlikely and unconventional lines of play where human tradition would suggest conceding.