Based on my observations, LLMs tend to be somewhat decent at:

• Writing small, well defined functions, or small sets of functions for which the input and outputs are fairly unambiguous
• Translating / porting code from language A to B
• Analyze binary formats, reverse engineer minified source code
• Writing average looking boilerplate UIs
• Optimizing code for which a goal metric is clearly established and a closed loop can be formulated and autonomously iterated
• Converting formats (e.g. rewrite Markdown to reStructuredText, JSON to CSV, etc.)
• Parse text and summarize / extract insights from such text

They are also very bad at:

• Autonomy, or being able to see the “bigger picture” on their own, even when provided with clear and unambiguous instructions to complete a full feature from A to Z.
• Decision making. More specifically they tend to have really poor taste when it comes to deciding to use a third party library vs. rewriting from scratch, choice of architectures, dependency vs. duplication, verbosity vs. ease, color styles, UX

It seems to me that the last two points, autonomy and decision making, are largely problematic for LLMs. On the autonomy, perhaps in a few years we will get there, as more capable models and more specialized agents are introduced, but the decision making part I’m not so sure. “Good decisions” are mostly opinions which are acquired over decades of real world experience (and pain) of trying other “not so good” (but perfectly working) decisions, but since LLMs train on the world archives, the best these tools can aspire to get are “mediocre decisions”. And maybe that will be enough for some.