How has economic evolution changed the way birthday cakes are made? In their classical Harvard business review article, Joseph Pine IIand James H. Gilmore asked this question and gave their answer:

1. In an agrarian economy, parents make birthday cakes by themselves. 
2. In an industrial economy, parents purchase premixed ingredients from the market.
3. In a service economy, parents order cakes as a yearly service for their kids.
4. In an experience economy, there will be “experience vendors” who could create an awesome birthday party for kids. 

Their framework is also applicable to software.

1. Agrarian economy: every business needs to write software for their use cases.
2. Industrial economy: there are software vendors dedicated to building software, which could be purchased by customers to solve their needs.
3. Service Economy: The software is delivered as a service (often as a subscription) and it would still require customers to build upon the service to create an awesome experience. This is the Software-as-a-Service (SaaS) model. 
4. Experience Economy: We are entering a new era of Software-as-a-Experience (SaaE). In the SaaE model, we use software to directly orchestrate experiences for our users. Software is not considered only a utility, but a conduit of digital experiences.

SaaE is a fundamental leap forward for the SaaS business model because it significantly reduces the friction for creating awesome user experiences. According to Joseph and James, 

an experience occurs when a company intentionally uses services as the stage, and goods as props, to engage individual customers in a way that creates a memorable event. …  No two people can have the same experience, because each experience derives from the interaction between the staged event (like a theatrical play) and the individual’s state of mind.

In the SaaS model, the software service itself is rigid. There is a predetermined way, often determined during the product build stage, to use the software. Users would continue their subscription only if the problem they face could be solved by the predetermined software flow. When this is no longer the case, customers would churn.

In the SaaE model, the software would be adaptive rather than rigid. The ultimate goal is to deliver the best experience to users by staging a sequence of software services intentionally.

To reach the goal, we need two new software layers:

1. A feedback-collection layer that could continually integrate the feedback from each customer to the product.
2. A learning layer for software behaviors to adjust the software services to deliver the best customer experience.

The two layers were not practical in the software building process before because it would take an enormous amount of effort to personalize the software flow for each user.

The advancement of AI has made personalizing unique experiences for each user possible. We have already done it — YouTube’s personalized feed creates a unique video watching experience for every user, and Amazon’s product recommendation has created a unique purchase experience for every customer. 

Those systems are currently referred to as recommendation systems. But this is just the tip of the iceberg for a large paradigm shift – the advent of the “Software-as-a-Experience” age. 

Morris Chang may not be a household name in the western world but his achievement is comparable to that of western business Titans like Rockefeller and Carnegie. His life consists of one miracle by another. After fleeing from China to the US during the China Civil War, he worked his way up to be the general manager of the Semiconductor businesses and 3rd ranked person of Texas Instrument — one of the biggest semiconductors companies in the world. Chang was one of the first Chinese Americans to become top business leaders. In his 50s, he returned to Taiwan and founded Taiwan Semiconductor Manufacturing Company (TMSC) and became the “godfather” of Taiwan’s semiconductor industry.

Chang was born in China in 1931. Most of his early life was deeply shaped by war — he was forced to flee three times due to the Second Sino-Japanese war and China civil war. After the wars, he came to the United States at the age of 17 to study at Harvard University, a school that primarily focused on arts and humanities education. As an underrepresented minority ethnic group, most Chinese Americans at the time worked in low-end restaurants or laundry businesses. Academic jobs were the only few alternative options that would fit with Chang’s mission of achieving big societal impacts. He later transferred to MIT in the hope of becoming a scholar in engineering majors. Unfortunately, he failed the qualifying exams twice at MIT and had to quit his academic path and went to the job market after obtaining his master’s degree. 

He entered the Semiconductor industry in his first job at Sylvania, an industry leader then. However, his team was dismantled three years later. He then moved to Texas Instrument, which was famous for the invention of integrated circuits (IC) and was fast-growing. After working at Texas Instrument for three years, he went back to Stanford to pursue a Ph.D. degree and then returned to Texas Instrument to continue his corporate duty. By the 1970s, Chang was already the general manager of the whole semiconductor business of Texas Instrument. 

Chang was a keen observer of the semiconductor industry. While working at Texas Instrument, he observed that a lot of brilliant people in the company were hoping to create new businesses but heavy investment requirements prevented them from getting started. As chips became more and more sophisticated, the chip manufacturing business became super capital-intensive. The cost of creating a chip manufacturing line (also known as “foundry” or “fab”) could easily be over 3-4 billion US dollars. Besides, new startups cannot maintain a sustainable stream of needs to keep their manufacturing line busy all the time, which is the only way to justify the heavy investments.

In contrast, chip designing requires much less capital. It would be a win-win situation if there is a “pure-play” company that focuses on manufacturing so that startups could focus on designing. This model of chip making process, also known as Fabless manufacturing as it features the split of designing and manufacturing, is crucial for the booming of the semiconductor industry. There were a lot of chip design talents in the US, but very few were good at both chip manufacturing and cost management. Chang was one of the few talents who had the expertise.

Chang started to face career setbacks in the early 1980s. At the time, Texas Instrument shifted focus away from semiconductors and became a diversified device manufacturer. Chang disagreed with the shift and had to leave the company. After the career setback, Change decided to turn his observation into action. At the same time, Taiwan government was eager to find ways to break into high-end industries like chip manufacturing. Chang was the perfect person to lead the cause. After a short stint at another company, Chang accepted the invitation from the Taiwan government to be the first chairman of the Industrial Technology Research Institute, an institute that played a critical role in the industrial transformation of the island. With the support of the Taiwan government, Chang founded the TSMC one year later. TSMC created a whole new industry of “pure-play” chip manufacturing (a.k.a., foundry industry). By focusing on only manufacturing but not designing, TSMC assured its partners, typically US chip designing firms, that TSMC won’t compete with them or share their trade secrets with their competitors. 

Now TSMC is undoubtedly the market leader in the industry and occupies 28% of the market share in a recent study. Also thanks to TSMC, Taiwan became crucial for the global semiconductor supply chain, which Bloomberg has recently published an article to illustrate.

Global semiconductor market share (image via Counterpoints Research)

In one of his recent talks, Chang gave a summary of what he thinks is the key reason for the TMSC’s success. Chang attributed the success to three factors. 

The number one is Taiwanese people’s hard-working spirit. For example, during his second stint at TMSC, Chang started the “nightingale program”, which included both day and night shifts to ensure there were R&D activities 24/7. This program would be unimaginable in U.S. companies. According to Chang, this nightingale program was the key reason why the TSMC could eclipse all of its competitors in technology. In chip manufacturing, the size of the device the manufacturing process could produce is a key indicator of technology level — smaller size means more devices in the same area but also is much harder to manufacture. After losing in the competition of the 14nm manufacturing process, TSMC reached the 10nm, 7nm, and 5nm manufacturing processes one by one in just a few years. Till now, none of its competitors have reached the 10nm milestone yet.  Please refer to RISC-V, China, Nightingales for more details.

Figure. The timeline of manufacturing process in major foundry companies.

The second factor is the local professional management. This factor is crucial because chip manufacturing is operation-heavy and efficiency-driven. Chang also mentioned that managerial talent doesn’t transfer well across borders because of culture and factors.

The third factor is the good infrastructure provided by the Taiwan government. It is easy to see how good infrastructure makes the transportation of goods much easier. Chang also mentioned an important point. The good high-speed railway and the small Island of Taiwan make it possible for talents to be relocated to any place within the island without the need to be separated from their families. The benefits of good infrastructure on the human management side are often ignored by governments but are crucial for businesses that require a lot of talent.

All the three points are about one thing: TSMC can attract a huge amount of disciplined and high-quality talent and can retain them through good management and providing a convenient life. The company has an envious 3-4% employee turnover rate. For those who leave for various reasons, they became the most sought-after talents in the industry. I also highly recommend this great essay by Kevin Xu based on Chang’s talk.

Throughout his life, Chang overcame one challenge by another and successfully turned setbacks into new opportunities. Although he became a refugee three times in his youth, he immigrated to the US to pursue a new life. After failing in MIT Ph.D. qualifying exams and crumbling his academic pursuit, Chang entered the newborn semiconductor industry and worked tirelessly to become an expert in semiconductor manufacturing. After facing a career setback in Texas Instrument, he took the courage to leave the US, a place he had spent 36 years, to Taiwan for creating TSMC and became the godfather of the semiconductor industry of the Island. 

Chang is also a great writer. I highly recommend his Chinese autobiography that covered his early life before 33 years old (unfortunately I haven’t found any translated version yet). Besides, he is working on the second half of this auto-biography, and hopefully, will publish it soon. I am very much looking forward to reading it and will share a sequel in the future.

For people who are interested in artificial intelligence, the past decade feels like another Renaissance — the boundaries between humans and machines are repeatedly re-defined by the invention of new technologies, from machines that could beat world Champions in games to AI assistants that could talk like real humans.

This article shares some stories behind this incredible AI Renaissance. The sources of the stories include my first-hand observation in the field as well as Cade Metz’s recent book  Genius Makers, which I highly recommend.

A Sputnik Moment

A key reason for the current revolution is the re-invention of deep learning, a technology that simulates human brains as complex network architectures through computers.

The idea is not new. Scientists have been searching for the truth of human intelligence for a long time, and a natural starting point is our brain — the only intelligent machinery built by our mother nature. Artificial neural networks, predecessors of deep learning, were very popular between the 1950s and 1980s but lost their popularity because data was not enough and the computers then were too weak to solve any interesting problems.

It would take another two decades before its revival. In the late 2000s, a group of young scientists started to connect the power of the booming Internet with artificial intelligence research. In 2009, an assistance professor of Princeton Dr. Fei-fei Li¹ compiled a vast database of Internet images (a.k.a. ImageNet dataset). The ImageNet dataset soon became the benchmark for Computer Vision, a subfield of artificial intelligence. In 2012, Geoffroy Hinton and his team significantly improved the metrics by more than ten percent, a jaw-dropping achievement that was a magnitude higher than any previous improvement.

This was a “Sputnik moment” for the artificial intelligence research community, at the time the mainstream research direction was for scientists to figure out the solution and to program the software based on the solution. Hinton’s success in the ImageNet challenge showed that an alternative approach — letting the neural networks learn a solution without prescription from humans — would work better. 

Godfather is Heading to Industry

Huge amounts of data and computer resources were essential to this success. No one knew this better than Geoffroy Hinton himself, who is also known as the Godfather of deep learning. Hinton was one of the early persons that popularized back-propagation, a fundamental algorithm used to train neural networks. When the field entered into a winter between the 1990s and early 2000s, most researchers switched to other research directions due to scarce funding sources. However, Hinton was still a stubborn proponent of the idea and was trying to revive it.

Hinton knew that his research would need resources from elsewhere, and only the big Internet companies had the pocket deep enough and data big enough to make the idea work. In addition to his academic achievement, Hinton also had great business savvy. With his two students, Hinton founded the DNNResearch company in 2012 and soon decided to sell it to big Internet companies. The book Genius Makers gave a vivid description of how Hinton orchestrated the auction in a Lake Tahoe hotel and how tech companies all over the world wooed him. DNNResearch was eventually acquired by Google for 44 million US dollars.

More important than the price tag is the precedence that Hinton created. To lure Hinton, Google allowed him to keep his position on both sides — but he had to be an “intern” in Google to work around the company’s rules. Before Hinton, it was rare for eminent researchers to work for tech companies because of the fear of losing their tenured positions in universities. Soon after the purchase, a lot of AI researchers followed Hinton’s example to join technology companies, including Yann LeCun,  another deep learning pioneer who later led Facebook’s AI lab, Andrew Ng, who led the research lab in Baidu. What’s more, following their advisors, students from various research labs flocked into big technology companies.

Among the technology companies, Google (and its parent company Alphabet) stood out for its unparalleled role in this wave of AI Renaissance. Its research divisions — Google Brain and DeepMind — are the driving force behind a lot of the greatest breakthroughs. What’s more, the fact that it could use AI to create so many profitable applications has a demonstration effect on all other companies.

One important person behind this is Jeff Dean, a legendary engineer that laid the foundation of Google’s infrastructure. In 2011, Andrew Ng introduced the deep learning concept to Jeff and he was intrigued immediately.  Jeff was looking for his next application and deep learning was a perfect one.  Andrew, Jeff, and another researcher Greg Corrado founded the Google Brain team. As a founding engineer of Google, Jeff has a great influence on Google’s management team and also enjoys enormous popularity within its engineering and research organizations (so much so that people made fun of him by creating the “Jeff Dean facts“). Jeff created an umbrella where the Google Brain team could operate without worrying about anything else.

In Google Brain, Andrew Ng and his colleagues helped create the system that could learn the “cat” concept from millions of YouTube videos, which drew a lot of media attention and publicized the field.  Andrew Ng eventually left the Brain team to work on his own startup. But he recommended Geoffroy Hinton as his replacement, which triggered the DNNResearch acquisition. Under the leadership of Jeff and Hinton, Google Brain significantly contributed to the field by both pushing the research frontiers and publishing the TensorFlow framework that makes the technology accessible to outside communities.

New World Champion

One limitation of the techniques Hinton was trying (a.k.a. supervised deep learning) was that it requires datasets labeled by humans. Demis Hassabis co-founded DeepMind to address this limitation and explore other applications. He wanted to build a system that doesn’t depend on human supervision and could perform better than humans. A child prodigy in Chess, Hassabis believes that games are the best starting points. Although games had been a proving ground for AI since the 50s, no one has been more committed and successful than Hassabis in this direction.

Hassabis and his DeepMind team combined deep learning with reinforcement learning, a technology that allows computers to adapt their behaviors through trial and error (the same way we humans learn). With this new technology, DeepMind built a system that could learn the nuances that were never found by humans before in popular video games like Breakout and published their results in Nature. This publication drew the attention of  Google’s management. In 2014, Google purchased DeepMind for more than $500M — this time both Hinton and Jeff are on the buyer side. With the resources from Google, DeepMind doubled down on its mission. In May 2017, DeepMind’s AlphaGo AI beat the world champion Ke Jie. Since then, it has kept beating humans in one field after another.  In addition to DeepMind, Google’s other AI division also released the BERT system that significantly improved performance in natural language tasks.

There were also a lot of breakthroughs outside Google and DeepMind. For example, OpenAI, which was co-founded by some of Hinton’s students and Silicon Valley elites like Elon Musk and YC CEO Sam Altman, tackled many other games and robotics applications through reinforcement learning and they released the language models that achieved amazing results. The successes of Google/DeepMind/OpenAI and other AI research teams have brought the public interest in AI to an unprecedented level.

What’s ahead?

A keen observer would find that the current AI Renaissance consists of many small cycles. Each cycle starts when a difficult yet well-defined benchmark problem is solved. Thanks to the huge public attention, the research team that solved the problem would be able to get a huge amount of resources to continue their research. The team then tackles the next more challenging benchmark problem with a larger model.  The cycles were started by academics and their students and were reinforced by big technology companies. People knew, either consciously or unconsciously, that it was the best way of attracting attention, funding, and talents.

Notwithstanding, ImageNet and Go games are still not real-world problems. In addition, there have been increasing concerns that this type of AI research pattern has caused enormous resource consumption and has made the AI models to be overly complex.For example, the GPT-3 language model related by OpenAI includes 175 billion parameters and each train takes around 4.6 million dollars.  In addition, many AIs that overfit man-made tasks turn out to perform poorly in many real-world applications.

We should and would break such cycles. Building cost-effective AI and making it really work in real-world applications is crucial to keep the movement going. In the next decade, there will be a lot more exciting stories ahead of us.

Disclaim: All opinions are mine and not endorsed by my current or previous employers.

---

1. Fei-fei Li was an assistant professor of Princeton University at the time but moved to Stanford later. The original version called Fei-fei Li a Stanford professor by mistake, thanks Jike Chong for pointing it out.

Eric Kandel won the Nobel prize in 2000 for his contribution to the understanding of memory at the molecular level. His autobiography, In Search of Memory, describes both his experience of escaping from Austria to America and his inquiry into the science behind our memory throughout his career. This essay is created based on the autobiography and extensive researches on the web.

Escape from Holocaust

Eric was born in a Jewish family in Vienna. As a Jew, Eric’s childhood was dominated by the Nazi’s growing influence in Austria. From its very beginning of the 1920s, the Nazi party aimed to merge all German-speaking people into a Greater Germany. In 1937, Hilter forced the Austrian chancellor Schuschnigg to resign and sent his troops to occupy the country, which was the largest German-speaking state outside Germany. The event, known as Anschluss, was welcomed by the Austrian Germans because a lot of them felt Austria was not fairly treated in the Treaty of Saint-Germain signed after WWI. After the Nazis took the power in Austria, a lot of Austrian Jews were forced to leave the country due to the violence targeting them (e.g., Kristallnacht). Thanks to the help of a local Jewish organization Kultusgemeinde, Eric’s family was able to emigrate to the United States of America in 1939. The young Eric was only Nine by then.

Most of the Jews who didn’t escape Nazi Austria became the victims of the Holocaust. The experience of escaping the Holocaust greatly influenced Eric throughout his whole life. Later after Eric won the Nobel prize in 2000, he used his influence to press the Austrian government to recognize the misfortune of the Jew community during Anschluss, which was largely ignored post-WWII, and to advocate the rights of Jews community in the country.

America

After emigrating to the US, Eric finished his education first in a Jewish school and then attended Harvard College. There Eric was attracted to psychoanalysis because it was imaginative, comprehensive, and empirically grounded. His attraction to psychoanalysis was further enhanced by the fact that its founder Freud was Viennese and Jewish and had been forced to leave Vienna. He later enrolled in New York University and aspired to become a psychoanalyst.  In the fall of 1955, Eric decided to take an elective at Columbia University with the neurophysiologist Harry Grundfest. Since then, Eric’s research career gradually shifted to find the biological basis of mental function.

Eric is particularly interested in the formation of memory. In 1890, William James concluded that memory must have at least two different processes: a short-term process and a long-term process. The basic units of the brain are the neurons, which are connected through synapses. Signals of one neuron are passed to the next neuron through chemical neurotransmitters that are available in synapse. One common hypothesis is that short-term memory is stored as the distribution of neurotransmitters across different synapses. A stimulus would activate a spatial pattern of activity across neurons in a brain region,  which will deplete the neurotransmitters. The distribution of neurotransmitters will form a trace of the stimuli, which is the short-term memory.

The short-term memory trace decays over time as neurotransmitters are re-generated. As a result, short-term memories need to be consolidated to long-term storage. Behavioral experiments suggest it happens through repetition — what is well known as “Practice makes perfect”.

Scientists also realized the importance of the hippocampus in turning short-term memory into long-term memory thanks to the extensive research on Henry Molaison (H.M.), who is probably the most famous patient in the history of memory research. After a treatment operation in which his hippocampus was removed, H.M.’s intelligence was intact, yet he lost the ability to form new memories. Other than this vague picture, Scientists had very little knowledge of the exact biochemical process of memory. It was under such a background that Eric entered the domain of memory research.

Aplysia

The first question Eric needs to answer is how neurons could adjust their connections based on environmental stimuli. Unfortunately, Human brains are too complex for any thorough analysis, each human brain has about 100 billion neurons. As a result, Eric experimented on Aplysia instead, whose brain has only about 20,000 cells, making it a perfect model animal to analyze how neurons work. In 1962,  Eric joined the lab of a French scientist Ladislav Tauc, one of the few scientists who worked on Aplysia then, as a post-doc to learn about this interesting sea slug. Eric’s work on Aplysia has laid the foundation for understanding the mechanism of memory — so much so that Eric presented a picture of Aplysia wearing a Nobel medal during his Nobel prize ceremony,

In Search of Memory

Eric and his team realized that long-term memories are formed as anatomical changes of the neurons. A single neuron has approximately 1300 presynaptic terminals (only 40% of which are active) with which it contacts about 25 different target cells. Through the consolidation process, the creates long-term memory, both the percentage of active presynaptic terminals and their total number. The number of synapses changes during learning. Memory is recalled when a certain sensory stimulus triggers the “reads out” of the new state of the synapse, which has been altered by learning.

In 1953, Waston and Crick proposed the famous Double Helix model of DNA, which opened the new world of molecular biology.  In the memory-research field, Louis Flexner from the University of Pennsylvania discovered that applying a drug that inhibits the synthesis of proteins would disrupt long-term memory. Eric realized that the same process also applies to Aplysia and that long-term memory storage requires the synthesis of new proteins.

One revolutionary breakthrough in molecular biology was the realization that gene function can be regulated up and down in response to environmental signals. Inspired by this breakthrough, Eric continued to investigate genes’ role in learning and memory formation.  Through researching Aplysia, Eric and his team realized that long-term memory is formed through switching on and off certain genes that increase or inhibit the growth of certain synapses.

For decades, Kandel has been studying how we create short-term and long-term memories at the molecular level. His work helps reveal the full picture of the memory-forming mechanism:

1. The memory storage takes place in at least two stages: A short-term memory lasting minutes is converted — by a process of consolidation that requires the synthesis of new protein — into stable, long-term memory lasting days, weeks, or even longer. 
2. A single stimulus strengthens the synapse through the depletion of neurotransmitters, which form the short-term memory.
3. Repeated stimulation causes certain genes to be switched on and the growth of new synapses, which creates long-term memory.

Eric’s journey from a refugee from Austria to a Nobel Laureate is a great example of how the tolerant and open environment of America could release boundless energy from immigrants like Eric and inspire them to think in new ways. In contrast, the city of Vienna, once a center of art and science, lost its glory under the suppressive occupation by the Nazis. His experience is still important for us after a hundred years.