What Is Everyday Computing

(Limits of LLMs and the Future of Every-Day Computers)

“If the organism carries a ‘small-scale model’ of external reality and of its possible actions within its head, it is able to try out various alternatives, conclude which is the best of them, react to future situations before they arise, utilise the knowledge of past events in dealing with the present and the future, and in every way to react in a much fuller, safer, and more competent manner to the emergencies which face it.” 

Kenneth Craik. Ch. 5 Hypothesis on the nature of thought

The Nature of Explanation” (1943)

Introduction

Current State of Smart Devices: Most smart devices today focus on individual interactions, which may need to fully capitalise on the potential for fostering community and collective experiences. These devices are designed primarily for personal use, which could contribute to isolated experiences rather than communal ones.

Vannevar Bush’s Memex project on screen-based interactions and the concept of the Internet has effectively meant that pieces of paper could be treated like magic. Every piece of paper could be connected to every other piece, and any information on that piece of paper could take you to any other. This idea could be transmuted. But I’m not only interested in paper as a design surface. Before screens, paper was just one way that we solved design problems. So, what do we do with the physical world? 

People live in an information-rich environment filled with natural and artificial objects extended across space and time. A wide variety of cognitive tasks - whether in everyday cognition, scientific practice, or professional life - require the processing of information across both the internal mind and the external environment. This interwoven internal and external information processing generates much of a person’s intelligent behaviour. The problems that I think the New Mother of All Demos will need to focus on are the representational issues in distributed cognitive tasks processed within our internal mind and the external environment, focusing on four problems: 

(a) The distributed representation of information; 

(b) The interaction between internal and external representations;  

(c) The nature of external representations;

(d) Human stigmergic problem-solving;

This idea is that the future of computation is in extending and augmenting human capabilities, an ancient idea about how to use computing. Still, it is through the sensing and actuation of objects and environments because the physical and digital are accurate and meaningful in our lives.

Closing the Context Gap

If you can understand a person's actions in their environment, you can predict needs based on their behaviour. If you can predict what someone will need in the subsequent moments, you could inform a new way to interact with technology. We already use this prediction when our GPS app tells us the best route around traffic. With AR glasses and the ability to layer three-dimensional imagery over existing spaces, we can build new interfaces that adapt to us as we switch activities, breaking free from the 2D interfaces we use today.  We call this new dynamic relationship with our digital world contextual computing. Contextual computing (n). 

Contextual AI: will be an all-day context-aware + all-day wearable that always uses machine perception + reasoning to predict possible outcomes. 

Predictive Models of Reality: If your twin carries a “small scale model” of external reality and of its possible actions within it, it can try out various alternatives, react to future situations before they arise, utilise the knowledge of past events in dealing with the present and the future, and in every way react in a much fuller, safer, and more competent manner to the emergencies that face it.  This could be anything from map directions on roads to x-ray views of our walls as we look for studs. Screens are irrelevant when all the spaces around you can hold the three-dimensional information that is most important to you.

Low Bandwidth

Language has low bandwidth: Less than 12 bytes/second. A person can read 270 words/minute or 4.5 words/second, which is 12 bytes/s (assuming 2 bytes per token and 0.75 words per token). A modern LLM is typically trained with 1x10^13 two-byte tokens, which is 2x10^13 bytes. This would take about 100,000 years for a person to read (12 hours a day).

Vision has a much higher bandwidth: About 20MB/s. Each of the two optical nerves has 1 million nerve fibres, each carrying about 10 bytes per second. A four-year-old child has been awake for 16,000 hours, translating into 1x10^15 bytes.

In other words, visual perception's data bandwidth: is roughly 1.6 million times higher than written (or spoken) language. In a mere four years, a child has seen 50 times more data than the biggest LLMs trained on all the text publicly available on the Internet.

This tells us three things: Yes, text is redundant, and visual signals in the optical nerves are even more redundant (despite being 100x compressed versions of the photoreceptor outputs in the retina). However, data redundancy is precisely what we need for self-supervised learning to capture the structure of the data. The more redundancy, the better for SSL. Most human knowledge (and almost all animal knowledge) comes from our sensory experience of the physical world. Language is the icing on the cake. We need the cake to support the icing. We will only reach human-level AI by getting machines to learn from high-bandwidth sensory inputs, such as vision. Yes, humans can get smart without vision, even pretty smart without vision and audition. But only with touch. Touch has high bandwidth, too.

To People Who Claim that "Thinking and Reasoning Require Language”.

Imagine standing at the North Pole of the Earth.

Walk in any direction, in a straight line for 1 km.

Now, turn 90 degrees to the left.

Walk in a straight line for as long as it takes to pass near the point where you turned.

Have you walked:

1. More than 2xPi km?

2. Exactly 2xPi km?

3. Less than 2xPi km?

4. I never came close to my starting point.

Think about how you tried to answer this question and tell us whether it was based on language.

Social Connection

Society needs to design a collaborative system focusing on the organisational structure that could enhance collective intelligence. If our new AI systems are contextually aware, we could apply the concept of stigmergy derived from social insects, which could be applied to human problem-solving and coordination. Considering all human interactions through stigmergy, there are insights into how decentralised coordination can lead to effective problem-solving and innovation. It mirrors the complex behaviours seen in the natural world. Still, applying it in a human context could help us design a new computing platform focused on a collaborative system that could enhance collective intelligence and efficiency.  Stigmergy in Human Problem-solving: Like insects, humans don't need complete solutions to contribute effectively. This is also seen where individuals can contribute based on existing progress and context, enabling collective problem-solving without centralised coordination. Applying stigmergy to the future of computing systems and AI could help develop new computing systems directly related to human interactions or behaviours within an environment. This could lead to more intelligent environments that anticipate our needs and facilitate positive behaviours, enhancing individual and community well-being. For example, an AI system could suggest optimal communal activities based on the historical patterns and preferences of the community members, promoting engagement and cooperation without direct human-to-human or human-to-AI interactions for each decision. New Computing Systems: The idea of a computing system that utilises traces of information or actions left in the environment is fascinating. This could lead to technology that understands individual needs and fosters communal behaviours and cooperative societies by encouraging and reinforcing positive actions and interactions. Future of Wearable Technology: The vision of wearables that understand our environment and provide contextual assistance is compelling. Such advancements could revolutionise how we interact with technology, making it more integrated into our lives and more supportive of our social behaviours and community interactions.

The Future of Computing

Currently, I like to think of AI as augmenting natural intelligence. With this, I wish to reduce the conversation about AI relative to humans and the labelling ‘better’, ‘worse’, ‘general’,’ narrow’, etc. What I do think is “Human + AI > Great Human.” Society’s progress is not just about one or a few knowledgeable individuals but also about the interconnect, the fabric, the "civilisation" part of human civilisation. What does a "Human + AI" society look like for "Human + Digital Companion > Great Human" to be true? Of course, what we mean by ">" or "better" is a variety of tasks, contexts, objectives, and granularities of the collective. It will need a lot of careful thought, but directionally, augmenting human intelligence at various granularities of a collective is more inspiring than calls to some individual, knowledgeable, artificial entity. 

The New Mother of All Demos 

Many brains work collectively, creating symbiotic relationships intersubjectively without being connected or understanding them. A super system whereby learning is transmitted from individuals to groups and communities around us, who then pass it on to others again, could also impact our inclusiveness of others. This needs collective thinking and empathy with people we have nothing in common with. But as we keep leaving behind trails of digital exhaust, we can examine and strive to understand the stigmergy and the science of swarm intelligence and the idea that your actions leave traces that signal your friends, family, or people how to interact and help each other. Can we move beyond a single interaction for a single person and think long-term over a day, week, month, and year? Who is the community you interact with daily, weekly, monthly and yearly? In summary, while technology is limited in fostering deep social connections and community building, the future holds the potential for more integrated, context-aware systems to support and enhance our social lives. The challenge will be to ensure these technologies are designed with these goals in mind, emphasising community, cooperation, and meaningful social interactions.

So, in the words of Isaac Asimov, “Your assumptions are your windows on the world. Scrub them off every once in a while, or the light won't come in" 

Thanks for Reading.

— Q