Generative AI: What It Does Well, and What It Doesn’t

Writer

Hillary Chabot

Hillary Chabot is a writer for Babson Thought & Action and Babson Magazine. An award-winning journalist, she is known for her insightful reporting and dedication to detailed storytelling. With a career spanning over two decades, she has covered a wide range of topics, from presidential campaigns and government policy to neighborhood issues and investigative series. As a reporter for The Boston Herald, Hillary earned a reputation for tenacity and integrity. Her work at Babson College fuels her passions—to learn something new every day and conduct thoughtful, empathic interviews. She’s thrilled to be at Babson College, where students, faculty, staff members and classes provide compelling copy daily.

August 29, 2024

Image of a person holding out their hand with the letters AI

Peter Cohan, an associate professor of practice in management, details generative AI's best uses and its limitations.

Listen

Generative AI has taken the technological world by storm, offering immense potential in content creation, design, and even decision-making processes. Babson College Associate Professor of Practice in Management Peter Cohan investigated the impact of generative AI in his new book, Brain Rush: How to Invest and Compete in the Real World of Generative AI. His research seeks to cut through portrayals of AI as either a utopian enabler or a dystopian disruptor, instead analyzing its current impact while looking for tangible evidence of value creation or potential pitfalls.

“I’m focused on what is happening right now in terms of generative AI,” Cohan said. “Is there anything that’s working, or is it all hype? Is there a return on investment in this? Is there any value being created or is this just another tech fad like Blockchain that fades away after people hype it for a few years?”

Contrary to fears of AI replacing jobs, Cohan encourages students to use and understand generative AI, emphasizing the importance of developing skills that complement AI capabilities. The limitations of AI in areas such as numerical and survey analysis highlight some of the skills that will remain indispensable.

“I start off all my classes by saying, ‘I want to future proof you. I don’t want you to be replaced by generative AI,’ ” Cohan said.

Cohan recently was named to RETHINK Retail’s inaugural list of Top AI Leaders in Retail for 2024. RETHINK Retail, a global leader in retail content and insights, named 150 leaders across 10 categories on the list for driving AI innovation. Cohan was named in the research and academia category.

Here, Cohan discusses what generative AI does well and what it doesn’t:

Generative AI Hits

Text-based generative AI, such as ChatGPT and Google’s Gemini, can help spark new ideas and streamline content quickly, Cohan said.

Peter Cohan, associate professor of practice in the Management Division, says generative AI struggles with analysis.

“It can come up with a list of ideas, and even though five of the ideas are obvious, there might be one that makes you think a little differently,” Cohan said. “It’s good for brainstorming in that way.”

The tool also excels at creating customer surveys, Cohan said. The management professor sometimes asks students to interview customers about a potential product they’re selling, then analyze those responses to find out if their startup creates enough value that customers will buy in.

With a few prompts, generative AI can easily create such a survey that will ensure students cover all the necessary marketing angles with consumers.

“It’s really good if you want to do a customer survey so you can decide whether a customer will buy from you or if they’ll buy from a competitor,” Cohan said. “It can come up with all the right questions in the right order.”

Generative AI Misses

Strangely enough, generative AI starts to reveal its weaknesses when it’s asked to analyze the answers to the customer survey it created, Cohan said.

“Generative AI doesn’t have a massive data storage of all these interviews, and it doesn’t know what customers want. It can’t even guess what customers want,” Cohan said.

Surprisingly, the analytical blind spot extends to numbers. More specifically, generative AI has issues analyzing, for example, a company’s annual report over the past five years to interpret what those numbers meant for the venture.

“I asked it to do things like produce a chart or go look at an annual report for a company and show me their revenues, their costs, and their profits. I’d ask it to detail how their financial statements have changed over time and explain why and where that company is going in the future,” Cohan explained.

“Basically, I was asking it to analyze a company. It can’t do any of that. It will maybe reprint part of the first page of a letter that the CEO wrote to shareholders, but it will not be able to penetrate any of the numbers or do any kind of quantitative analysis.”

While generative AI is constantly evolving, Cohan’s research offers a practical guide to navigating the complex landscape of one of the most significant technological advancements of our era.