I did tell it that I expect to see something like a power-law distribution in order value, so I think I pretty much followed your instructions here. Btw, if you do know the right thing to do in my scenario, I'd love to figure it out. This is not my area of expertise, and just figuring it out through articles so far.
I recommend reading Wikipedia and talking to LLMs to get this one. Order values do follow power-law distributions (you're probably looking for an exponential or a Zipf distribution.) You want to ask how to perform a statistical test using these distributions. I'm a fan of Bayesian techniques here, but it's up to you if you want to use a frequentist approach. If you can follow some basic calculus you can follow the math for constructing these statistical tests, if not some searching will help you find the formulas you need.
Thanks for the suggestions! I didn't want to do the math myself, but I did take your suggestion and found some articles discussing ways to make it work even with a non-normal distribution:
I'm not checking their math, but the articles make sense to me, and I trust they did implement it correctly. In the end the LLM did get me to the correct answer by suggesting the articles, so I guess I should eat some humble pie and say it _did_ help me. At the same time, if I didn't have the intuition that using rpv as-is in a t-test would be noisy, and the suggestions from this comment thread, I think I could have gone down the wrong path. So I'm not sure what my conclusion is -- maybe something like LLMs are helpful once you ask the right question.
One heuristic I like to use when thinking about this question (and I honestly wish the answer space here were less emotionally charged, so we could all learn from each other) is that: LLMs need a human to understand the shape of the solution to check the LLM's work. In fields that I have confirmed expertise in, I can easily nudge and steer the LLM and only skim its output quickly to know if it's right or wrong. In fields I don't, I first ask the LLM for resources (papers, textbooks, articles, etc) and familiarize myself with some initial literature first. I then work with the LLMs slowly to make a solution. I've found that to work well so far.
(I also just love statistics and think it's some of the most applicable math to everyday life in everything from bus arrival times to road traffic to order values to financial markets.)
