Increasing GenAI Project Success Rate with Strands Analyst
How to decide between using LLMs vs not based on industry, domain, use case, market segment and other factors. How to manage transition of mental models between traditional to probabilistic systems.
I was following the recent press coverage for an MIT report claiming that “95% of generative AI pilots at companies are failing”. The core issue? Not the quality of the AI models, but the “learning gap” for both tools and organizations, according to the coverage.
The “solution” came soon after as I read the awesome Building AI Products In The Probabilistic Era article from a researcher Gian Segato who is currently working at Anthropic and previously at Replit. The article goes into a fairly detailed explanation of differences in mental models between the deterministic (most software) and probabilistic era (AI systems).
This was a good use case for me to take help from Strands Analyst which I have been building as covered in my prior posts.
I turned to Strands Analyst to summarize the decision matrix for me.
Strands Analyst Prompt: Read https://giansegato.com/essays/probabilistic-era and create a markdown table of key insights comparing side by side deterministic and probabilistic, categorized by insights, to help in decision making when ensuring use case viability for one over the other and what it takes to succeed. Do an analysis by industry, domain, market segment, etc. Save as markdown. Create a high quality image version of the tables.
Strands Analyst created a markdown version of the analysis.
Then Strands Analyst turned to Python code generation to convert markdown tables to images.
Here is the overall summary of differences between deterministic and probabilistic systems. Primary mental model shift is reaching the Minimum Viable Intelligence threshold - “the lowest quality threshold that is both accepted by the market (some may be more sensitive than others), while preserving its inherent flexibility and generalization capacity”.
Here is the second comparison Strands Analyst generated to note impact on product development and engineering domain. Pivotal aspect here is system design which changes the mental model to empirical testing and willingness to (throwaway prior design and) rebuild.
For product management and design a critical aspect is user experience in probabilistic era which requires managing uncertainty of infinite ways in which user may end up using the product. This domain goes through the most significant mental model shifts across all aspects.
For organizational structure and data the major shift is in decision making based on probabilistic insights.
The technology and software industry segment undergoing most disruption right now include developer tools. Is enterprise software next?
Marketing domain saw the early adoption use cases. Insurance industry is seeing significant adoption now compared with others, most likely due to disproportionate document processing (claims) workflows which have already seen robotic process automation.
Healthcare and Life Sciences industry has many opportunities because of relatively higher dependence on empirical vs engineered outcomes.
Among use cases Search is seeing significant disruption thanks to Perplexity, Glean, and others.
Market segments seeing early adoption are obviously Startups.
On implementation strategy cost structure is seeing interesting trends around outcome based pricing across domains.
I also realized that I can turn the insights from this article into a scenario analysis tool for my customers. So, I used Strands Analyst again to turn the 4,500 words article into an interactive decision making tool within a few minutes using the following prompt!
Strands Analyst Prompt: Create a decision making app based on this article to help user do what if scenario analysis based on their needs across various categories of insights comparing deterministic and probabilistic systems, so that they can get the right recommendations.
This worked on a generated markdown version of the original article.
Within minutes I got an HTML and Python script variants of the decision making tool. Very cool use case of Situational User Interface generation!
