Categories
English

Build your technology foundation in FAST way to embrace the Generative AI revolution.

We are currently in the midst of a boom era for Generative AI, and the landscape is changing almost daily. For more details, please refer to this blog post. The possibilities seem endless!

However, this is also an increasingly confusing time for builders. After spending months fine-tuning GPT-3, you may wake up to find that the model is now irrelevant due to the availability of GPT-4. Similarly, while working on building your workflow, you may discover that AutoGPT has taken over your niche.

In this fast-changing era, it’s crucial to build a solid foundation that allows you to leverage the benefits of the generative AI revolution.

FAST Framework

We introduce the FAST Framework for describing the characteristics of a tech team suitable for building a generative AI product. The FAST Framework consists of four elements:

  1. Flexibility: To support product pivots without wasting a lot of work.
  2. Accuracy: To ensure that AI generates valuable results for users.
  3. Scalability: To enable a good user experience at a large scale and cost-effectively.
  4. Tight-knit development workflow: To allow the team to build products with user feedback in a high-velocity manner.

How Storytell Plans to build our Technology Foundations

Let me explain how Storytell plans to incorporate the framework. To begin, let me describe the Storytell system at a high level. Our technology stack primarily consists of three systems:

  1. The front-end system, which includes anything related to user interface and user interaction.
  2. The back-end system, which handles data processing.
  3. The AI system, which is our core algorithm for distilling signal from noise.

Flexibility

For the front-end system, one of the key challenges is to be flexible enough to adapt to different distribution channels such as the Chrome Web Store, Apple App Store, websites, and Slack App. Each channel has its own front-end UI development guideline. However, as the current AI era represents a paradigm shift, it is unclear which channel will eventually emerge as the winner. Therefore, it is essential to be able to experiment quickly at this stage. This often means revamping the front-end UI every few months. It would be a huge waste of effort if a team had to start from scratch each time they need to prioritize a new distribution channel.

The critical challenge for AI systems is to design an architecture that is agnostic to changes in the generative AI platform. For an overview of the generative AI platform, please see this A16z blog.

Accuracy

Accuracy has been a major issue for Large Language Models (LLMs), which are the workhorses of generative AI. LLMs from Google, OpenAI, and Microsoft all suffer from “hallucinations,” in which the AI confidently generates fake statements. Accuracy is a significant obstacle to applying LLMs to high-value use cases, such as enterprise applications. Any company working in the generative AI field must have a systematic way to improve accuracy of their AI system.

Two capabilities are required to address accuracy problems.

  1. Fast user feedback and adaptive learning. Users should have the ability to report inaccuracies, and the system should be able to quickly adapt to feedback.
  2. System to Ensure Accuracy: An AI system should adhere to certain rules or guarantees. One important rule is that the output should never be inappropriate. Additionally, due to the probabilistic nature of LLMs, it is necessary to build a software stack on top of them to ensure accuracy.

Scalability

Generative AI can be both expensive and slow. While APIs like OpenAI and open source models like LLaMA have made it easier to build generative AI systems, it remains challenging to build a scalable back-end system. The fast-changing landscape of generative AI makes this particularly difficult.

  1. If you choose to use an API, the latency is often very high. Therefore, you need to design a system that can accommodate this latency. On the other hand, if you choose to use open-source models, the accuracy is likely to be worse. In this case, you need to design a system that can improve accuracy, for example, by using certain types of fine-tuning. Additionally, you need to design your system to tune efficiently because the base models change very rapidly.
  2. Many API and open-source models have limits on the maximum amount of text you can send to the API, known as the token limit. To ensure that your system works within this limit and does not negatively impact your model’s performance, you need to design it accordingly.
  3. Many API providers specify a maximum quota for API calls. You will need to work within this requirement by minimizing the number of API calls made.

Tight-knit development workflow

The typical product development cycle involves the following steps: user feedback collection -> feature prioritization -> development -> feature impact analysis. It is crucial to accelerate this cycle and make it as continuous as possible.

  1. When collecting user feedback, you may have a separate customer team responsible for its collection. However, it is crucial for your engineers to also listen to user feedback. This requires having a development team that is highly focused on the needs of users.
  2. Having a systematic way to identify important features that will benefit the business is crucial for feature prioritization. It’s essential to align the entire team with a single metric, known as the north-star metric. Prioritization should be done on a daily basis, rather than quarterly.
  3. For software development, it is essential to be able to ship code as quickly as possible. Both technology capabilities and market needs are evolving rapidly. To find a product-market fit, you need to move faster than both. This means you need to be able to ship code continuously. Investment in software development lifecycle (SDLC) tools and continuous integration/continuous delivery (CI/CD) systems is crucial.
  4. For feature impact analysis, it is important to collect user data as early as possible. This allows you to analyze user patterns and identify the top use cases. It is also worthwhile to invest in the business intelligence stack early on.

We are currently hiring for our Backend/ML role. If Storytell’s vision resonates with you and you want to help build it, please email me at jingconan@storytell.ai. For more information, please visit our career page and vision statement.

Categories
English

SaaS is Dead. Long Live SaaE

How has economic evolution changed the way birthday cakes are made? In their classical Harvard business review article, Joseph Pine IIand James H. Gilmore asked this question and gave their answer:

  1. In an agrarian economy, parents make birthday cakes by themselves. 
  2. In an industrial economy, parents purchase premixed ingredients from the market.
  3. In a service economy, parents order cakes as a yearly service for their kids.
  4. In an experience economy, there will be “experience vendors” who could create an awesome birthday party for kids. 

Their framework is also applicable to software.

  1. Agrarian economy: every business needs to write software for their use cases.
  2. Industrial economy: there are software vendors dedicated to building software, which could be purchased by customers to solve their needs.
  3. Service Economy: The software is delivered as a service (often as a subscription) and it would still require customers to build upon the service to create an awesome experience. This is the Software-as-a-Service (SaaS) model. 
  4. Experience Economy: We are entering a new era of Software-as-a-Experience (SaaE). In the SaaE model, we use software to directly orchestrate experiences for our users. Software is not considered only a utility, but a conduit of digital experiences.

SaaE is a fundamental leap forward for the SaaS business model because it significantly reduces the friction for creating awesome user experiences. According to Joseph and James

an experience occurs when a company intentionally uses services as the stage, and goods as props, to engage individual customers in a way that creates a memorable event. …  No two people can have the same experience, because each experience derives from the interaction between the staged event (like a theatrical play) and the individual’s state of mind.

In the SaaS model, the software service itself is rigid. There is a predetermined way, often determined during the product build stage, to use the software. Users would continue their subscription only if the problem they face could be solved by the predetermined software flow. When this is no longer the case, customers would churn.

In the SaaE model, the software would be adaptive rather than rigid. The ultimate goal is to deliver the best experience to users by staging a sequence of software services intentionally.

To reach the goal, we need two new software layers:

  1. A feedback-collection layer that could continually integrate the feedback from each customer to the product.
  2. A learning layer for software behaviors to adjust the software services to deliver the best customer experience.
Figure: Two layers of the SaaE.

The two layers were not practical in the software building process before because it would take an enormous amount of effort to personalize the software flow for each user.

The advancement of AI has made personalizing unique experiences for each user possible. We have already done it — YouTube’s personalized feed creates a unique video watching experience for every user, and Amazon’s product recommendation has created a unique purchase experience for every customer. 

Those systems are currently referred to as recommendation systems. But this is just the tip of the iceberg for a large paradigm shift – the advent of the “Software-as-a-Experience” age. 

Categories
English

Morris Chang: From Refugee to the Godfather of Taiwan’s Semiconductor Industry

Morris Chang may not be a household name in the western world but his achievement is comparable to that of western business Titans like Rockefeller and Carnegie. His life consists of one miracle by another. After fleeing from China to the US during the China Civil War, he worked his way up to be the general manager of the Semiconductor businesses and 3rd ranked person of Texas Instrument — one of the biggest semiconductors companies in the world. Chang was one of the first Chinese Americans to become top business leaders. In his 50s, he returned to Taiwan and founded Taiwan Semiconductor Manufacturing Company (TMSC) and became the “godfather” of Taiwan’s semiconductor industry.

Chang was born in China in 1931. Most of his early life was deeply shaped by war — he was forced to flee three times due to the Second Sino-Japanese war and China civil war. After the wars, he came to the United States at the age of 17 to study at Harvard University, a school that primarily focused on arts and humanities education. As an underrepresented minority ethnic group, most Chinese Americans at the time worked in low-end restaurants or laundry businesses. Academic jobs were the only few alternative options that would fit with Chang’s mission of achieving big societal impacts. He later transferred to MIT in the hope of becoming a scholar in engineering majors. Unfortunately, he failed the qualifying exams twice at MIT and had to quit his academic path and went to the job market after obtaining his master’s degree. 

He entered the Semiconductor industry in his first job at Sylvania, an industry leader then. However, his team was dismantled three years later. He then moved to Texas Instrument, which was famous for the invention of integrated circuits (IC) and was fast-growing. After working at Texas Instrument for three years, he went back to Stanford to pursue a Ph.D. degree and then returned to Texas Instrument to continue his corporate duty. By the 1970s, Chang was already the general manager of the whole semiconductor business of Texas Instrument. 

Chang was a keen observer of the semiconductor industry. While working at Texas Instrument, he observed that a lot of brilliant people in the company were hoping to create new businesses but heavy investment requirements prevented them from getting started. As chips became more and more sophisticated, the chip manufacturing business became super capital-intensive. The cost of creating a chip manufacturing line (also known as “foundry” or “fab”) could easily be over 3-4 billion US dollars. Besides, new startups cannot maintain a sustainable stream of needs to keep their manufacturing line busy all the time, which is the only way to justify the heavy investments.

In contrast, chip designing requires much less capital. It would be a win-win situation if there is a “pure-play” company that focuses on manufacturing so that startups could focus on designing. This model of chip making process, also known as Fabless manufacturing as it features the split of designing and manufacturing, is crucial for the booming of the semiconductor industry. There were a lot of chip design talents in the US, but very few were good at both chip manufacturing and cost management. Chang was one of the few talents who had the expertise.

Chang started to face career setbacks in the early 1980s. At the time, Texas Instrument shifted focus away from semiconductors and became a diversified device manufacturer. Chang disagreed with the shift and had to leave the company. After the career setback, Change decided to turn his observation into action. At the same time, Taiwan government was eager to find ways to break into high-end industries like chip manufacturing. Chang was the perfect person to lead the cause. After a short stint at another company, Chang accepted the invitation from the Taiwan government to be the first chairman of the Industrial Technology Research Institute, an institute that played a critical role in the industrial transformation of the island. With the support of the Taiwan government, Chang founded the TSMC one year later. TSMC created a whole new industry of “pure-play” chip manufacturing (a.k.a., foundry industry). By focusing on only manufacturing but not designing, TSMC assured its partners, typically US chip designing firms, that TSMC won’t compete with them or share their trade secrets with their competitors. 

Now TSMC is undoubtedly the market leader in the industry and occupies 28% of the market share in a recent study. Also thanks to TSMC, Taiwan became crucial for the global semiconductor supply chain, which Bloomberg has recently published an article to illustrate.

Global Semiconductor Market 2020.png

Global semiconductor market share (image via Counterpoints Research)

In one of his recent talks, Chang gave a summary of what he thinks is the key reason for the TMSC’s success. Chang attributed the success to three factors. 

The number one is Taiwanese people’s hard-working spirit. For example, during his second stint at TMSC, Chang started the “nightingale program”, which included both day and night shifts to ensure there were R&D activities 24/7. This program would be unimaginable in U.S. companies. According to Chang, this nightingale program was the key reason why the TSMC could eclipse all of its competitors in technology. In chip manufacturing, the size of the device the manufacturing process could produce is a key indicator of technology level — smaller size means more devices in the same area but also is much harder to manufacture. After losing in the competition of the 14nm manufacturing process, TSMC reached the 10nm, 7nm, and 5nm manufacturing processes one by one in just a few years. Till now, none of its competitors have reached the 10nm milestone yet.  Please refer to RISC-V, China, Nightingales for more details.

Figure. The timeline of manufacturing process in major foundry companies.

The second factor is the local professional management. This factor is crucial because chip manufacturing is operation-heavy and efficiency-driven. Chang also mentioned that managerial talent doesn’t transfer well across borders because of culture and factors.

The third factor is the good infrastructure provided by the Taiwan government. It is easy to see how good infrastructure makes the transportation of goods much easier. Chang also mentioned an important point. The good high-speed railway and the small Island of Taiwan make it possible for talents to be relocated to any place within the island without the need to be separated from their families. The benefits of good infrastructure on the human management side are often ignored by governments but are crucial for businesses that require a lot of talent.

All the three points are about one thing: TSMC can attract a huge amount of disciplined and high-quality talent and can retain them through good management and providing a convenient life. The company has an envious 3-4% employee turnover rate. For those who leave for various reasons, they became the most sought-after talents in the industry. I also highly recommend this great essay by Kevin Xu based on Chang’s talk.

Throughout his life, Chang overcame one challenge by another and successfully turned setbacks into new opportunities. Although he became a refugee three times in his youth, he immigrated to the US to pursue a new life. After failing in MIT Ph.D. qualifying exams and crumbling his academic pursuit, Chang entered the newborn semiconductor industry and worked tirelessly to become an expert in semiconductor manufacturing. After facing a career setback in Texas Instrument, he took the courage to leave the US, a place he had spent 36 years, to Taiwan for creating TSMC and became the godfather of the semiconductor industry of the Island. 

Chang is also a great writer. I highly recommend his Chinese autobiography that covered his early life before 33 years old (unfortunately I haven’t found any translated version yet). Besides, he is working on the second half of this auto-biography, and hopefully, will publish it soon. I am very much looking forward to reading it and will share a sequel in the future.

Categories
English

Renaissance of Intelligence

For people who are interested in artificial intelligence, the past decade feels like another Renaissance — the boundaries between humans and machines are repeatedly re-defined by the invention of new technologies, from machines that could beat world Champions in games to AI assistants that could talk like real humans.

This article shares some stories behind this incredible AI Renaissance. The sources of the stories include my first-hand observation in the field as well as Cade Metz’s recent book  Genius Makers, which I highly recommend.

A Sputnik Moment

A key reason for the current revolution is the re-invention of deep learning, a technology that simulates human brains as complex network architectures through computers.

The idea is not new. Scientists have been searching for the truth of human intelligence for a long time, and a natural starting point is our brain — the only intelligent machinery built by our mother nature. Artificial neural networks, predecessors of deep learning, were very popular between the 1950s and 1980s but lost their popularity because data was not enough and the computers then were too weak to solve any interesting problems.

It would take another two decades before its revival. In the late 2000s, a group of young scientists started to connect the power of the booming Internet with artificial intelligence research. In 2009, an assistance professor of Princeton Dr. Fei-fei Li1 compiled a vast database of Internet images (a.k.a. ImageNet dataset). The ImageNet dataset soon became the benchmark for Computer Vision, a subfield of artificial intelligence. In 2012, Geoffroy Hinton and his team significantly improved the metrics by more than ten percent, a jaw-dropping achievement that was a magnitude higher than any previous improvement.

This was a “Sputnik moment” for the artificial intelligence research community, at the time the mainstream research direction was for scientists to figure out the solution and to program the software based on the solution. Hinton’s success in the ImageNet challenge showed that an alternative approach — letting the neural networks learn a solution without prescription from humans — would work better. 

Godfather is Heading to Industry

Huge amounts of data and computer resources were essential to this success. No one knew this better than Geoffroy Hinton himself, who is also known as the Godfather of deep learning. Hinton was one of the early persons that popularized back-propagation, a fundamental algorithm used to train neural networks. When the field entered into a winter between the 1990s and early 2000s, most researchers switched to other research directions due to scarce funding sources. However, Hinton was still a stubborn proponent of the idea and was trying to revive it.

Hinton knew that his research would need resources from elsewhere, and only the big Internet companies had the pocket deep enough and data big enough to make the idea work. In addition to his academic achievement, Hinton also had great business savvy. With his two students, Hinton founded the DNNResearch company in 2012 and soon decided to sell it to big Internet companies. The book Genius Makers gave a vivid description of how Hinton orchestrated the auction in a Lake Tahoe hotel and how tech companies all over the world wooed him. DNNResearch was eventually acquired by Google for 44 million US dollars.

More important than the price tag is the precedence that Hinton created. To lure Hinton, Google allowed him to keep his position on both sides — but he had to be an “intern” in Google to work around the company’s rules. Before Hinton, it was rare for eminent researchers to work for tech companies because of the fear of losing their tenured positions in universities. Soon after the purchase, a lot of AI researchers followed Hinton’s example to join technology companies, including Yann LeCun,  another deep learning pioneer who later led Facebook’s AI lab, Andrew Ng, who led the research lab in Baidu. What’s more, following their advisors, students from various research labs flocked into big technology companies.

Among the technology companies, Google (and its parent company Alphabet) stood out for its unparalleled role in this wave of AI Renaissance. Its research divisions — Google Brain and DeepMind — are the driving force behind a lot of the greatest breakthroughs. What’s more, the fact that it could use AI to create so many profitable applications has a demonstration effect on all other companies.

One important person behind this is Jeff Dean, a legendary engineer that laid the foundation of Google’s infrastructure. In 2011, Andrew Ng introduced the deep learning concept to Jeff and he was intrigued immediately.  Jeff was looking for his next application and deep learning was a perfect one.  Andrew, Jeff, and another researcher Greg Corrado founded the Google Brain team. As a founding engineer of Google, Jeff has a great influence on Google’s management team and also enjoys enormous popularity within its engineering and research organizations (so much so that people made fun of him by creating the “Jeff Dean facts“). Jeff created an umbrella where the Google Brain team could operate without worrying about anything else.

In Google Brain, Andrew Ng and his colleagues helped create the system that could learn the “cat” concept from millions of YouTube videos, which drew a lot of media attention and publicized the field.  Andrew Ng eventually left the Brain team to work on his own startup. But he recommended Geoffroy Hinton as his replacement, which triggered the DNNResearch acquisition. Under the leadership of Jeff and Hinton, Google Brain significantly contributed to the field by both pushing the research frontiers and publishing the TensorFlow framework that makes the technology accessible to outside communities.

New World Champion

One limitation of the techniques Hinton was trying (a.k.a. supervised deep learning) was that it requires datasets labeled by humans. Demis Hassabis co-founded DeepMind to address this limitation and explore other applications. He wanted to build a system that doesn’t depend on human supervision and could perform better than humans. A child prodigy in Chess, Hassabis believes that games are the best starting points. Although games had been a proving ground for AI since the 50s, no one has been more committed and successful than Hassabis in this direction.

Hassabis and his DeepMind team combined deep learning with reinforcement learning, a technology that allows computers to adapt their behaviors through trial and error (the same way we humans learn). With this new technology, DeepMind built a system that could learn the nuances that were never found by humans before in popular video games like Breakout and published their results in Nature. This publication drew the attention of  Google’s management. In 2014, Google purchased DeepMind for more than $500M — this time both Hinton and Jeff are on the buyer side. With the resources from Google, DeepMind doubled down on its mission. In May 2017, DeepMind’s AlphaGo AI beat the world champion Ke Jie. Since then, it has kept beating humans in one field after another.  In addition to DeepMind, Google’s other AI division also released the BERT system that significantly improved performance in natural language tasks.

There were also a lot of breakthroughs outside Google and DeepMind. For example, OpenAI, which was co-founded by some of Hinton’s students and Silicon Valley elites like Elon Musk and YC CEO Sam Altman, tackled many other games and robotics applications through reinforcement learning and they released the language models that achieved amazing results. The successes of Google/DeepMind/OpenAI and other AI research teams have brought the public interest in AI to an unprecedented level.

What’s ahead?

A keen observer would find that the current AI Renaissance consists of many small cycles. Each cycle starts when a difficult yet well-defined benchmark problem is solved. Thanks to the huge public attention, the research team that solved the problem would be able to get a huge amount of resources to continue their research. The team then tackles the next more challenging benchmark problem with a larger model.  The cycles were started by academics and their students and were reinforced by big technology companies. People knew, either consciously or unconsciously, that it was the best way of attracting attention, funding, and talents.

Notwithstanding, ImageNet and Go games are still not real-world problems. In addition, there have been increasing concerns that this type of AI research pattern has caused enormous resource consumption and has made the AI models to be overly complex.For example, the GPT-3 language model related by OpenAI includes 175 billion parameters and each train takes around 4.6 million dollars.  In addition, many AIs that overfit man-made tasks turn out to perform poorly in many real-world applications.

We should and would break such cycles. Building cost-effective AI and making it really work in real-world applications is crucial to keep the movement going. In the next decade, there will be a lot more exciting stories ahead of us.

Disclaim: All opinions are mine and not endorsed by my current or previous employers.


  1. Fei-fei Li was an assistant professor of Princeton University at the time but moved to Stanford later. The original version called Fei-fei Li a Stanford professor by mistake, thanks Jike Chong for pointing it out.
Categories
English

In Search of Memory

Eric Kandel won the Nobel prize in 2000 for his contribution to the understanding of memory at the molecular level. His autobiography, In Search of Memory, describes both his experience of escaping from Austria to America and his inquiry into the science behind our memory throughout his career. This essay is created based on the autobiography and extensive researches on the web.

Escape from Holocaust

Eric was born in a Jewish family in Vienna. As a Jew, Eric’s childhood was dominated by the Nazi’s growing influence in Austria. From its very beginning of the 1920s, the Nazi party aimed to merge all German-speaking people into a Greater Germany. In 1937, Hilter forced the Austrian chancellor Schuschnigg to resign and sent his troops to occupy the country, which was the largest German-speaking state outside Germany. The event, known as Anschluss, was welcomed by the Austrian Germans because a lot of them felt Austria was not fairly treated in the Treaty of Saint-Germain signed after WWI. After the Nazis took the power in Austria, a lot of Austrian Jews were forced to leave the country due to the violence targeting them (e.g., Kristallnacht). Thanks to the help of a local Jewish organization Kultusgemeinde, Eric’s family was able to emigrate to the United States of America in 1939. The young Eric was only Nine by then.

Hitler announces the Anschluss on the Heldenplatz, Vienna, 15 March 1938.

Most of the Jews who didn’t escape Nazi Austria became the victims of the Holocaust. The experience of escaping the Holocaust greatly influenced Eric throughout his whole life. Later after Eric won the Nobel prize in 2000, he used his influence to press the Austrian government to recognize the misfortune of the Jew community during Anschluss, which was largely ignored post-WWII, and to advocate the rights of Jews community in the country.

America

After emigrating to the US, Eric finished his education first in a Jewish school and then attended Harvard College. There Eric was attracted to psychoanalysis because it was imaginative, comprehensive, and empirically grounded. His attraction to psychoanalysis was further enhanced by the fact that its founder Freud was Viennese and Jewish and had been forced to leave Vienna. He later enrolled in New York University and aspired to become a psychoanalyst.  In the fall of 1955, Eric decided to take an elective at Columbia University with the neurophysiologist Harry Grundfest. Since then, Eric’s research career gradually shifted to find the biological basis of mental function.

Eric is particularly interested in the formation of memory. In 1890, William James concluded that memory must have at least two different processes: a short-term process and a long-term process. The basic units of the brain are the neurons, which are connected through synapses. Signals of one neuron are passed to the next neuron through chemical neurotransmitters that are available in synapse. One common hypothesis is that short-term memory is stored as the distribution of neurotransmitters across different synapses. A stimulus would activate a spatial pattern of activity across neurons in a brain region,  which will deplete the neurotransmitters. The distribution of neurotransmitters will form a trace of the stimuli, which is the short-term memory.

The short-term memory trace decays over time as neurotransmitters are re-generated. As a result, short-term memories need to be consolidated to long-term storage. Behavioral experiments suggest it happens through repetition — what is well known as “Practice makes perfect”.

Scientists also realized the importance of the hippocampus in turning short-term memory into long-term memory thanks to the extensive research on Henry Molaison (H.M.), who is probably the most famous patient in the history of memory research. After a treatment operation in which his hippocampus was removed, H.M.’s intelligence was intact, yet he lost the ability to form new memories. Other than this vague picture, Scientists had very little knowledge of the exact biochemical process of memory. It was under such a background that Eric entered the domain of memory research.

Most of Molaison’s two hippocampi were removed bilaterally.

Aplysia

The first question Eric needs to answer is how neurons could adjust their connections based on environmental stimuli. Unfortunately, Human brains are too complex for any thorough analysis, each human brain has about 100 billion neurons. As a result, Eric experimented on Aplysia instead, whose brain has only about 20,000 cells, making it a perfect model animal to analyze how neurons work. In 1962,  Eric joined the lab of a French scientist Ladislav Tauc, one of the few scientists who worked on Aplysia then, as a post-doc to learn about this interesting sea slug. Eric’s work on Aplysia has laid the foundation for understanding the mechanism of memory — so much so that Eric presented a picture of Aplysia wearing a Nobel medal during his Nobel prize ceremony,

“Aplysia Won the Nobel Prize”

In Search of Memory

Eric and his team realized that long-term memories are formed as anatomical changes of the neurons. A single neuron has approximately 1300 presynaptic terminals (only 40% of which are active) with which it contacts about 25 different target cells. Through the consolidation process, the creates long-term memory, both the percentage of active presynaptic terminals and their total number. The number of synapses changes during learning. Memory is recalled when a certain sensory stimulus triggers the “reads out” of the new state of the synapse, which has been altered by learning.

In 1953, Waston and Crick proposed the famous Double Helix model of DNA, which opened the new world of molecular biology.  In the memory-research field, Louis Flexner from the University of Pennsylvania discovered that applying a drug that inhibits the synthesis of proteins would disrupt long-term memory. Eric realized that the same process also applies to Aplysia and that long-term memory storage requires the synthesis of new proteins.

One revolutionary breakthrough in molecular biology was the realization that gene function can be regulated up and down in response to environmental signals. Inspired by this breakthrough, Eric continued to investigate genes’ role in learning and memory formation.  Through researching Aplysia, Eric and his team realized that long-term memory is formed through switching on and off certain genes that increase or inhibit the growth of certain synapses.

For decades, Kandel has been studying how we create short-term and long-term memories at the molecular level. His work helps reveal the full picture of the memory-forming mechanism:

  1. The memory storage takes place in at least two stages: A short-term memory lasting minutes is converted — by a process of consolidation that requires the synthesis of new protein — into stable, long-term memory lasting days, weeks, or even longer. 
  2. A single stimulus strengthens the synapse through the depletion of neurotransmitters, which form the short-term memory.
  3. Repeated stimulation causes certain genes to be switched on and the growth of new synapses, which creates long-term memory.

Eric’s journey from a refugee from Austria to a Nobel Laureate is a great example of how the tolerant and open environment of America could release boundless energy from immigrants like Eric and inspire them to think in new ways. In contrast, the city of Vienna, once a center of art and science, lost its glory under the suppressive occupation by the Nazis. His experience is still important for us after a hundred years.

Categories
English

What modern business leaders could learn from Genghis Khan?

Genghis Khan as portrayed in a 14th-century Yuan era album;

Genghis Khan created the Mongol Empire, the biggest empire in human history. At its height, the Mongol Empire covered a land area of more than 9.15 million square miles and a population of more than 100 million. Another surprising fact is that the population of Mongols was only a few million.

Why were Mongols able to conquer the world with such a tiny population? One important reason is that Genghis Khan created a specialized organization that could leverage the most advanced technology at the time (Mongolian horses) to solve the most ambitious problem (conquering the world). 

Despite the small number of human soldiers, there were a huge number of horses in the Mongol army. Each Mongol soldier has 3-4 Mongolian horses at his disposal at any time. Mongolian horses had very great endurance and were the most advanced military technologies during the cold-weapon era. In contrast, their enemies either don’t have any horses or could only use inferior horses. 

More importantly, Genghis Khan organized his Mongol soldiers in a way that could leverage the advantages of those Mongolian horses to the full extent. The command structure of the Mongol army was much more flexible than other armies during the period. Lower-level leaders have significant license to execute orders in the way they considered best. The super flexible organization allowed Mogol armies to attack en masse, divide into smaller groups to encircle and lead enemies into an ambush, or divide into small groups to mop up a fleeing and broken army. Because they could fully leverage the mobility of horses, a few Mongolian cavalry soldiers could easily defeat hundreds of foot soldiers.

Thanks to horses, the Mongolian army could cover up to 100 miles (160 km) per day, which was unheard of by other armies of the time. Mongolian soldiers were able to travel thousands of miles without stopping by rotating horses during the trip. Because of such great mobility, the Mongol empire could allocate resources on a global scale to defeat every local enemy. For example, the Mongols were able to fight with both the Muslim world and China at the same time. After Mongols conquered Muslims, they were able to leverage the technology they got from Muslims (like the counterweight trebuchet) to destroy the Song dynasty.

Genghis Khan and his Mongolian armies have taught us two things:

  1. New technology requires a new form of human organization to fully leverages its power. 
  2. An organization that could leverage the new power would be able to unlock even more new opportunities.

In the past few decades, we are creating new technologies to extend our brains. One notable new technology is artificial intelligence (AI), which allows machines to make predictions and decisions autonomously.  The relationship between the new AI tools and humans is similar to horses and Mongolian soldiers.

A business would need to transform its organizational structure to fully leverage the power of AI tools.

  1. For a lot of traditional businesses, the bottom of the organizational chart is a huge number of employees who work on operational tasks. As a result, management is based on carrots and sticks. More advantage management like (motivation alignment) is only available for strategic positions.
  2. In AI-first organizations, even junior employees will have hundreds of AI tools at his/her disposal and their influence on the organization is equivalent to a much higher-level person in those traditional organizations. Organizational management needs to be more motivation-driven throughout the organization.  The organization also (is able to ) and needs to be leaner and flatter, which encourages innovation.
The hidden workforce of AI-first organizations

Proactively leveraging AI tools not only reduces cost but also unleashes new powers (like horses do to Genghis Khan’s troops).  

  1. The natural way of organizational growth is to throw hiring humans. However, more people would create a communication burden and operational overhead. As an organization grows, the Return-On-Investment (ROI) of extra hiring will eventually decrease to be below 1, which prevents the company to scale further. 
  2. “Hiring” AI systems, in contrast, would not incur extra overhead. What’s more, AI systems typically get smarter as more people use them. As a result, the ROI will increase as the usage of the AI system increases.

The only ceiling floor for the scaling of an AI system is from the technical side. Currently, most of the commercial-viable AI system is only designed for a single problem. And for most of the problems, AI systems haven’t reached the human-level yet. This will be a bottleneck in the foreseeable future but more and more AI systems will be invented as time goes by. Human + AI collaboration would be a strong disruptive power for industries in which AI solutions are available. 

Hiring more people doesn’t make the manager’s job redundant. Instead, it makes their jobs more important. Similarly, the adoption of AI systems doesn’t make their users redundant. They will increase the scope of their users and the whole organization. Humans are tremendously flexible and could always find creative new usage of new capabilities.  For example, AI may be able to help doctors to diagnose basic medical conditions, but it won’t be able to replace doctors. Instead, doctors would be able to focus on more complicated medical problems. As long as humans haven’t reached immortality, there are always new problems for doctors to solve.

We don’t want another Mongol empire that causes deaths, but we do need business growth that could make human life better.  In addition to scaling the human part of the organization, every business leader should also consider where their “horses” are and how to provide organizational support to enable employees to use them.

Reference

  1. Mongol military tactics and organization
  2. Wikipedia: Mongol military tactics and organization.
  3. “The Mongol Empire’s Best Weapon: The Mongolian Horse” History on the Net © 2000-2021, Salem Media.
Categories
English Newsletter

Conan’s Newsletter No. 16

Book of the Week: Shoe Dog

I typically recommend a book only if it is worth reading multiple times. Shoe Dog by Nike co-founder is one of such books. I finished reading this book a few months back when my manager recommended it and completed another pass when I was on a road trip to Death Valley. Every read gives me fresh thoughts because Phil is a great storyteller, and Shoe Dog is not a typical memoir.

The book starts with a vivid description of Phil’s world trip right before he founded Nike. In the 1960s, German products dominated the American sports shoe market. As a former track runner at college, Phil envisioned that Japanese running shoes would become significant competitors to German shoes. During his stop in Japan, he made contact with a Japanese shoemaker Onikusa. When Phil returned from his word trip with a contract to distribute Onitsuka shoes in the U.S., he started his legendary Nike journey.

In the early days of Nike, Cashflow has always been a bottleneck for its growth. Although the American venture capital was booming then, most of them were in Silicon Valley, far away from Nike’s headquarter in Portland. Besides, the Shoe business is not the high-growth field V.C.s were looking for. As a result, Nike had to grow from bootstrapping and from bank loans. For the first five years, Phil had to keep a day job to earn Nike cash and work on Nike in the evenings and weekends. Nike was continuously groaned by its bankers and almost fell into bankruptcy in 1975.  

Nike rides the tide of Globalization. From Nike, you could see how Globalization (particularly Japan) have profoundly influenced the United States in the 1970s. Nike was initially just a distributor of Onikusa in the USA. Later, when it started selling shoes, it relied on loans from Nissho, a Japanese trade company.

Flying geese paradigm

The history of Nike is also a history of supply chain outsourcing from the U.S. to east Asia. Despite that it is the most famous sports shoe brand globally, Nike itself manufactures nothing and entirely relies on a global supply chain, which gave Nike the edge over Adidas. Nike’s supply chain was first in Japan, then was moved to Taiwan and later to China. Nike is not alone. A lot of other American companies (such as Apple and Tesla) followed the pattern.

Japanese scholar Akamatsu’s came up with a concept of Flying geese paradigm for the phenomenon that the Asian countries would catch up with the West like flying geese because the production of commoditized goods would continuously move from the more advanced countries to the less advanced ones in the region hierarchy. 

The flying geese paradigm is the reason why there are so many economic miracles in east Asian in the past few decades (Japanese economic miracleMiracle on the Han RiverTaiwan Miracle)

Then why Asia? Some crucial reasons are the region’s social and cultural characteristics: hard-working ethic, Collectivism, and low labor cost (initially), which are somewhat very different and complementary with the Western culture. You could get more context for the difference in this excellent documentary American Factory.

However, the social and cultural characteristics that help the countries catch up with the West are a double-edged sword. Although the flying geese paradigm created economic miracles in those countries, it also makes them prone to the “technology snapshot trap,”— a phenomenon in which a society develops involutely in a “snapshot” of the outdated technology because it fails to learn from outside or generate innovation innately continuously.

For example, Japan developed an advanced automobile and Electronics industry in the 1970s but failed to lead the personal computer revolution. Korea and Taiwan picked up the semiconductor industry but missed the Internet. Recently, “Involution” also became a hot topic in China social media. More and more people complain that society starts to stagnate and more and people have to face more fierce competition on limited resources.

In east Asia, the working population suffers from severe over-working (e.g., 996Karoshi) in the catchup process. Over-work culture will prevent people from learning new things and reduces fertility rates, which will drive up labor costs and reduce the competitive advantage of society in the long run. The obedient culture in the region also reduces the diversity of ideas and disruptive innovation within itself.

Companies like Nike combine both ends’ advantages by leveraging the West’s marketing and sales creativity and delivering high-quality yet cheap products using the Asia supply chain. However, this fundamentally drives the tension on both ends and is the inherent reason for a couple of trade wars.

We are facing a dilemma for Globalization. American people complain about the loss of manufacturing jobs, and Asian countries complain that the West captures most profits. We are at the crossroad of deglobalization, and the pandemic adds fuel to the process. A healthy society needs to strive for the right balance between the two cultures. Both ends should learn more from each other and take the opportunity to transform their culture and industry structures.

Categories
English Newsletter

The Revenge of Apple to Intel

One hot topic recently is that Apple released its new ARM chips — M1. It is not the first time Apple designs chips — Apple has successfully designed chips for its iPhone and IPads. It is also not the first time Apple uses non-Intel chips in its Mac products — Mac had Intel cores only since 2006.

Then why is it important? In short, this is a declaration of war from Apple to Intel and a game-changer for Reduced Instruction Set Computer (RISC) in performance-sensitive applications.

What are Instruction Sets?

Developers use chip instruction sets to communicate with computer chips. Metaphorically chip instruction sets are similar to the alphabets of human languages.

There are only twenty-six characters in English, but more than three thousand in Chinese. Similarly, the size of chip instruction sets also varies. Reduced Instruction Set Computer (RISC) refers to building chips using a small instruction set. In contrast, Complex Instruction Set Computer (CISC) refers to the option that uses an extensive instruction set. (Please see here for more descriptions)

A little history

Early computer chips were all CISC and mostly were designed by Intel. In the 1980s, there was a movement of reducing the instruction set. The ARM technology was founded in this period, and Apple–IBM–Motorola alliance built the PowerPC chips for Macintosh computers.

On the other side of the table, the Windows-Intel alliance (a.k.a “WinTel“) kept investing heavily in CISC. The rest is history; WinTel crushed Apple computers in personal computing. Apple had to switch to Intel chips in 2006. ARM survived only in a then niche market of IoT devices thanks to its energy efficiency.

Then the mobile Internet era came, thanks to Apple’s iPhone release. ARM is appealing for those applications because people care about the battery life of smartphones. As a result, ARM captured 90% of the market share for mobile processors. Intel lost the mobile war because it suffered from the Innovator’s Dilemma and wasn’t willing to risk upsetting its existing CISC business.

Despite ARM’s success in mobile phones, Intel still holds the crown for applications that require high-performance. Many people think this is due to CISC’s inherent superiority in high-performance computation, and Intel is safe in those fields.

Apple declared this is wrong through the release of M1. Intel maintained CISC’s advantage in the high-performance applications through massive investment, and previously there was no significant player who could compete.

Except for Apple. Some early users mention the performance of M1 could be comparable to NVIDIA’s popular 1080Ti GPU. The TensorFlow team also shows new M1 chips could outperform many workstations for AI applications, which have the highest computation requirements. 

What’s more, Apple has a great track record for disrupting industries. A lot of ARM manufacturers will follow Apple’s path to optimize ARM for high-performance applications, and they are eager to do so, given that the mobile phone market is saturating.

Besides, NVIDIA now owns ARM. The merger gives both edges in the age of AI. The road ahead for Intel is not rosy. Would the aging Titan be able to hold its position? It’s hard to say. But one thing is sure. More competition in the field is a great thing for companies in downstream areas like Cloud and AI, which could benefit from increased computation powers and reduced cost.

The market share ARM in different fields

Note: There is an interesting podcast from A16z about Apple Silicon. 16 Minutes #46: Apple Silicon — A Long Game, Changing the Game

Note: Although it is very promising, please still wait for a few months before you decide to upgrade to Big Sur or M1 chip if you want to use it for ML training. A lot of the libraries are not compatible with the new system yet (tweet)

Categories
English

What AI practitioners could learn from Tesla

This is the second blog about Tesla, please also read the blog of The Rise and Fall of a Great Inventor if you are interested to learn more about Tesla’s life.

Tesla is one of the key figures in the early evolution of the electrical industry. Tesla has good showmanship and is very good at attracting public attention through jaw-dropping demos.  In one such public demo, Tesla ignited light bulbs using his body. Those demos helped Tesla raise funding for his Alternating-Current motors, which greatly extended the applications of electricity.

0_tc81proTFGVIpWqC.pngTesla’s Magnifying transmitter 

In Tesla’s later years, his focus shifted to wireless energy transmission. Tesla planned to set up a set of energy transmission towers in the world, and any person could receive energy through a hand-held device. It was a grand project. Tesla raised some initial funding from J.P. Morgan to implement a prototype. Unfortunately, an Italian physicist and radio pioneer Marconi finished the wireless telegraph across the Atlantic Ocean in 1901, which attracted most of the public attention and overshadowed Tesla’s work. What’s worse, Tesla spent all the funding to build a huge tower in Wardenclyffe but failed to deliver a workable solution. He was turned down when trying to request more funding from J.P. Morgan. He was never able to fulfill this dream for the rest of his life.

0_5ASkdPah11dXRjfb

1904 Image of Wardenclyffe Tower.

Although it happened one century ago, Tesla’s story is still very relevant in the contemporary world in which AI is the new electricity. As an AI practitioner, I think there are several lessons we could learn from Tesla’s experience.

First, even for a super ambitious project, it is still important to make sure there are reasonable deliverables in the process. An ambitious vision may be crucial to get the initial resources. But in order to keep the marathon running, it is always good to plan a sequence of deliverables throughout the journey. The anti-pattern of promising too much while delivering too little needs to be avoided. Tesla was a visionary Inventor, but he lacked the practical mindset to manage the expectation of investors and showing deliverables.

Second, it is super important to be mindful of the relevant opportunities and be flexible for the plan. The development of technology is never a linear process. Tesla’s technology was very similar to what Marconi used for telegraph across the Atlantic and Tesla had much more experience than Marconi. Why didn’t he become the inventor of the telegraph? He failed to realize another important application of his technology — information transmission — and went straight to the grand goal of wireless energy transmission. Had he realize that achieving wireless communication was equally important and may be helpful for his final goal, he probably would invest more in the direction. 

For AI, the 2016 game between AlphaGo and Lee Sedol played a similar role as Tesla’s public demonstrations. The game attracted huge public attention and made many people realize the potential of AI. Under this hype, a lot of companies were founded with super ambitious goals that require decades to fulfill. And a lot of investors invested without a good understanding of this. What’s worse, a lot of the companies didn’t set up reasonable deliverables in a typical cycle of an investment fund. When those investors realized this gap, they may pull back investments, which will make the industry enter another winter.

It doesn’t mean that we shouldn’t work on moonshot AI projects. On the one hand, a lot of advanced AI projects will and should take place in universities under public support. On the other hand, for AI moonshots that are done in companies, we need to balance the grand vision with concrete milestones that are associated with the company’s core business. For example, one of the fields that AI works very well so far is the recommender system (e.g. the algorithm behind and Youtube or Instagram Feeds). The main reason for this is that its deliverables are very quantifiable (e.g., improve the daily activities users by x percentage) and directly contribute to the core business of the company, which is crucial to ensure continual support. I hope other fields could also find a similar positive feedback loop. It won’t be an easy path, but it is something that industrial AI  practitioners need to figure out.

Categories
English

The Rise and Fall of a Great Inventor

Recently I finished the book “Tesla: Inventor of Electrical Age” by W. Bernard Carlson and I highly recommend this book. I will write multiple blogs about the book and this first one will focus on historical facts and my thoughts on the rise and fall of Tesla.

tesla.jpg

Nikola Tesla was born in 1856 in the Austra-Hungarian empire. He came to the US in June 1884 to work for Edison Machine Works and left the company after a short-stay of 6 months. Later, he was approached by businessmen Benjamin A. Vail and Robert Lane to form a company but the two persons abandoned Telsa after a year. In the fall of 1886, Tesla was rescued by two other business partners Peck and Brown, who underwrote Tesla’s efforts to develop inventions into practical devices. On July 7, 1888, Peck and Brown sold the Tesla patents to Westinghouse for a lucrative deal and Tesla started to serve as a consultant for Westinghouse. Tesla left Westinghouse in August 1889 and in 1891 Westington tore apart the contract with Tesla under the pressure of investors. From 1892, Tesla started to give consultation on the Niagara fall powerhouse project, which finished in 1895 and established Tesla’s reputation as one of America’s leading inventors.

During 1895 and 1898, Tesla investigated some other things, like X-ray and radio-controlled boats. In 1899 and 1900, Tesla stayed in Colorado to perfect the wireless transmission system. In Nov 1900, Tesla was able to meet with the most powerful man on Wall Street, J.P. Morgan, and convinced Morgan to loan him $150,000 to support his wireless network. Around the same time, the Italian inventor Marconi was also working on a competing technology. In Dec 1901, Marconi finished the transmission of Telegraphy through Atlantic. The loss of the competition with Marconi forced Tesla to bet all-in on an even bolder project of wireless transmission of power. After the project failed in 1905, the life of Tesla as a bold inventor came as an end. Tesla spent most of this remaining life as a recluse in a New York hotel and was forgotten soon.

Telsa’s inventor career could be divided into three stages: Rise, Plateau, and Downfall stages. The first phase (Rise) is from when Tesla started to work for the Edison company and ended at the time when his sponsor Peck died. In this stage, Tesla, as a young immigrant, challenged the industry with his innovative thoughts of AC transmission. Tesla built his reputation by showing many magical demonstrations of electricity. Despite the dramatic promotion, Tesla’s work at this stage was pretty practical.

After Peck’s death that marked the start of the second stage (plateau), Tesla struggled to create a project that had commercial potential. Different from the first stage, Tesla at this time had more resources and still did a lot of amazing demonstrations to the public. However, a lot of his efforts like X-ray and Radio-controlled boats went nowhere. The biggest achievement this time is Niagara fall. However, Tesla’s role in the project is only a consultant and this is more a continuation of his work in the first stage. The lack of evidence to bring his innovative ideas to concrete commercial success restricted his ability to find patrons. 

The third stage is when Tesla got into the building the wireless energy transmission. He was able to secure some funding from patrons (like Astron and JP morgan) to pursue his dream thanks to his fame. However, none of his patrons at this stage was as serious and devoted as Peck and Brown were. To make matter worse, Tesla had to make bolder and bolder claims because he couldn’t catch up with new innovators like Marconi. The claims he made (like establishing a global wireless energy network) eventually backfired and cost him the credibility that was essential for him to raise funding.

Like everything in this world, the rise and the fall of Tesla are likely to be caused by many factors. I want to mention two of them in this blog:

A fundamental reason is that Tesla changed from a challenger to be a defender.  The rise of Tesla is because of his great contribution to Alternating Current (AC) technology. When Tesla first came to the US, Direct Current (DC) was more popular because it had a headstart in both research and industry. Notably, Edison was an ardent supporter of DC. However, AC had technical advantages over DC for long-distance transmission of electricity. Edison probably also realized the potential of AC. However, as the stakeholder of the Edison electric company (later became GE), which had already invested heavily in DC, Edison had to defend his commercial interests. In contrast, As a penniless immigrant, Tesla had no such burden so he chose to focus on the less-popular AC technology. In addition, like Steve Jobs, Tesla had the ability to create a Reality distortion field around him and to change people’s views. For example, Peck and Brown initially wanted Tesla to focus on DC that already had a market. Tesla organized a dramatic demonstration of using AC to make a copper egg spin by themselves that turned Peck and Brown into ardent supporters of AC. Through those demonstrations and continuous improvement of the technology, Tesla successfully challenged the status of DC.

The situation became completely different when Tesla was competing with Marconi for wireless technology. Similar to Edison in the 1980s, Tesla has been blinded by the sunk cost. Tesla’s long-term success in using electricity as a medium of energy transmission made him unable to realize the significance of wireless communication. Although he did propose a plan of using his technology for information transmission, it was mostly a strategy to secure funding and energy was still the main focus. Eventually, Tesla failed in the competition because of the complexity of wireless energy transmission. The newcomer Marconi didn’t have this burden and set wireless communication as the primary focus from day one. 

Another reason is that Tesla couldn’t find another strong business partner to fill the gap after Peck’s death. It was Peck who helped Tesla set up the strategy of patent-promote-sell that Tesla used throughout his career. However, the recipe wouldn’t work without any of the three ingredients. Tesla is very good at innovating and patenting. However, Tesla lacks the business acumen to execute the promotion and sales strategies.

Undoubtedly Tesla has a great talent for showmanship. However, the key to promotion in this context is to establish credibility among the professionals and managers in the electrical industry, who are the decision-makers for Tesla’s patents. Peck knew it very well so he tried to secure the endorsement of Professor Anthony, a well-established figure in the community, as the first step of the promotion campaign. After Peck passed away, Tesla relied mostly on mass media, which eventually portrayed him as a magician instead of a serious inventor. The mass media coverage helped Tesla in the beginning but eventually backfired and made him harder to secure financial support. Tesla also lacked sales and negotiation skills. For example, Peck helped Tesla negotiate the deal with Westinghouse, which was very favorable to Tesla himself, but Tesla allowed Westinghouse to tear it apart in 1891 after Peck passed away. Later, Tesla negotiated a very unfavorable deal with J.P. Morgan, which allowed Morgan to take the majority stake without a clear clarification of Morgan’s duty in the partnership. The ambiguity eventually damaged the partnership and the deal became a blocking stone when Tesla tried to raise funding from other investors.

Despite the enormous legacy he has left us, Tesla was forgotten for a long time. It was only in recent years that he re-entered people’s attention thanks to the electric car brand named after him. Interestingly, the founder of Tesla Motors, Elon Musk, shares a lot of characteristics with Tesla. Both are bold innovators and are good at showmanship. As a great disruptive innovator, Tesla and his story are still relevant in our contemporary world.