Extreme AI Expert at Three Months: Lessons, Insights, and Innovations

In my first blog of this series, I introduced the launch of Extreme AI Expert, a groundbreaking tool that debuted in July 2024 as a phased technical preview for Extreme employees and partners. Designed to empower IT teams, Extreme AI Expert streamlines the process of identifying and resolving network issues, making it easier to run, optimize, and monitor networks. By harnessing the power of generative AI, the tool analyzes an extensive database of technical documents to deliver precise, tailored solutions to network-related challenges.

My second blog explored the initial insights and lessons learned during the first month of the technical preview. We learned valuable lessons about user engagement, feedback collection, and system accuracy. Gamification strategies, regional competitions, and consistent communication drove remarkable participation, exceeding our initial goals and providing us with structured feedback mechanisms to gather actionable insights to refine the system. We discovered that fostering authentic user interactions and continuously improving the accuracy of responses are key to building trust and driving adoption. Additionally, early metrics showed a growing reliance on Extreme AI Expert as users integrated it into their daily workflows.

Building on the data foundation of technical documentation, we have entered an exciting new phase of the preview by integrating operational data into the knowledge base. This integration of real-time network environments has unlocked a deeper level of system capability, enabling us to deliver context-specific recommendations that transcend the limitations of static technical documents. Through this advancement, we are learning valuable lessons about how real-world data enhances the system’s ability to provide actionable insights tailored to dynamic network conditions. These new lessons are shaping the evolution of Extreme AI Expert, further refining its ability to address complex scenarios with precision and relevance

Now, after three months, we achieved significant improvements, increasing positive feedback rates to the point where 97% of interactions are now neutral or positive—approaching the desired level of accuracy comparable to human performance. This milestone is crucial as we effectively cut the number of negative feedback instances in half.

However, as we progress, our focus is shifting beyond traditional QA to adopting implicit feedback mechanisms for more accurate and effective quality measurement. Tools like sentiment analysis enable us to evaluate responses without relying on explicit feedback, which users often bypass or dismiss—similar to ignoring or hastily clicking through five-star ratings in a Microsoft Teams call. While explicit user feedback can be helpful, it’s frequently superficial, serving more to dismiss prompts than as genuine input.

Therefore, to gain deeper understanding, we are closely analyzing conversational user behaviors like rephrasing or clarification attempts, abrupt conversation stops, and sudden topic changes. All of these behaviors are essential measures of how smoothly and effectively users engage with the system. And despite a low overall inaccuracy rate of 1.57% (with only 0.23% in follow-up queries), there is still room for improvement. These behaviors serve as critical indicators of the system's fluidity and effectiveness, highlighting areas where users may encounter friction or require additional clarity. By studying these patterns, we can identify opportunities to improve the system's ability to anticipate user needs, deliver seamless experiences, and foster more intuitive interactions.

Additionally, we have determined that consistent communication with users is critical because expectations are constantly evolving. For instance, during a period of over three weeks without a new release, we noticed a decline in adoption rates in certain areas, as activity dropped when users perceived stagnation. The rapid pace of innovation in generative AI creates an expectation for frequent updates and new features, almost on a weekly basis. To meet these demands, Extreme Networks is staying ahead by aligning with this dynamic industry rhythm. Our approach combines proprietary methodologies together with customer operational data to deliver a level of accuracy and relevance far beyond off-the-shelf generative AI solutions like OpenAI’s ChatGPT or Gemini. This tailored strategy not only differentiates us but also helps us address the unique and evolving needs of our users effectively.

For instance, a comparison of responses using available technical documentation highlights the superior accuracy and detail provided by Extreme AI Expert. When I asked the high-level question, "What are the top features of the Extreme AP5020?"—ChatGPT delivered a correct but less comprehensive answer, whereas Extreme AI Expert offered a far more detailed and nuanced response.

Additionally, while ChatGPT referenced general sources from the Extreme Networks website, Extreme AI Expert drew from more pertinent resources, such as the installation guide and data sheets, to provide a more focused and relevant response.

When I asked each platform direct questions about configuration in ExtremeCloud IQ, I found that Extreme AI Expert referred to the most up-to-date 24.4.0 version of the ExtremeCloud IQ User Guide, while ChatGPT relied on the older 23.4.0 version. This highlights a key advantage of Extreme AI Expert: its proprietary methodologies help ensure more accurate document management and the ability to query the latest and most relevant resources, even when much of Extreme's documentation is publicly available.

As mentioned earlier in this blog, we are now using operational data that is drawn from a network environment. I asked Extreme AI Expert to show me by percentage the top operating system of the connected Wi-Fi clients in the past 24 hours and received this response:

When asking ChatGPT the same question the response was as expected. This is not a knock on ChatGPT, but a public system simply does not have access to the operational data in real-time.

Extreme AI Expert has come a long way in just three months, setting a strong foundation for the future of network and security operations. By integrating real-time operational data, leveraging proprietary methodologies, and analyzing user behaviors, we’ve not only improved system accuracy but also enabled it to adapt seamlessly to the evolving needs of IT teams. This approach provides the foundation for augmented analytics and creates a new experience in how users interact with and gain insights from data. In a democratized approach, there is no longer a need to be, or involve, a data expert.

The lessons learned during this journey—whether from user feedback, engagement strategies, or performance comparisons—have been instrumental in refining the tool. As we continue to innovate, our focus remains on empowering IT teams with precise, actionable insights that drive efficiency and simplify operations. The journey is far from over, and we eagerly anticipate what lies ahead as we continue to enhance the capabilities of Extreme AI Expert. Future advancements include double-click AI agents, multi-agent architectures, and agentic workflows—innovations that will unlock new use cases and achieve levels of accuracy beyond what is possible with a single large language model (LLM) alone.