406
AI bots hallucinate software packages and devs download them
(www.theregister.com)
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
this post was submitted on 28 Mar 2024
406 points (100.0% liked)
Technology
59334 readers
4252 users here now
This is a most excellent place for technology news and articles.
Our Rules
- Follow the lemmy.world rules.
- Only tech related content.
- Be excellent to each another!
- Mod approved content bots can post up to 10 articles per day.
- Threads asking for personal tech support may be deleted.
- Politics threads may be removed.
- No memes allowed as posts, OK to post as comments.
- Only approved bots from the list below, to ask if your bot can be added please contact us.
- Check for duplicates before posting, duplicates may be removed
Approved Bots
founded 1 year ago
MODERATORS
I just want an LLM with a reasonable context window so we can actually write real working packages with it.
The demos look great, but it’s always just around 100 lines of code, which is beginner level. The only use case right now is fake packages.
Just use the AI Horde. iirc our standard is like 4K context and some people host up to 8K. Here's a frontend
8k context is nothing.
Claude is 200k
those are tokens not lines of code....
??? The top lvl commenter wants an LLM with big context window and the other commenter responded with an LLM which has 200k token context window which is waaaaaay more than "100 lines of code".
Yeah sorry, I thought that was clear. It’s how context is measured.
I use it for writing functions a lot, tell it the inputs and desired outputs it'll normally make what i want. Recently gpt has got good at continuing where it left off too.
I'm using Codeium for that. Works pretty well as a glorified autocomplete, but not much more. Certainly saves a lot of typing though, but I have to double-check everything it produces, because sometimes it adds subtle errors.
I have tried the copilot integration in edge out of curiosity, and if you feed the ai the context of the page the response can be useful. There is a catch, tho:
when opening a document the accepted formats are html, txt, pdf. The documentation of a software package can be summarized but thr source will be the context of the page and not a web search, which is good in this casr
when generating new information, the model can be far too sintethic, cutting out potentially useful informations.
I still think you need to read the documentation yourself, maybe using the AI integration only when you need a general idea of the document.
What I do is first reading the summary of the documebt by bullet point, than reading the pdf file as a whole. By the time I do so, the LLM has given enough of a structure to facilitate my readings...
I'm not particularly interested. Some on my team are playing with it, but I honestly don't see much point since they spend more time fixing the generated code than they would writing it.
And I don't think it'll ever really work well (in the near-ish future) for the most common type of dev work: fixing bugs and making small changes to existing code.
It would be awesome if there was some kind of super linter instead. I spend far more time reading code than writing it, so if it can catch bugs, that would be interesting.
In my experience with Codeium, it sometimes works ok for three or four lines of code at once. I've actually had a few surprises where it nailed what I was going for where I didn't expect it. But most of the time, it's just duplicating code from elsewhere in the same file, which usually doesn't make sense.
It's also pretty good for stuff where I'd usually build some exotic regex to search/replace (or do it manually, because it'd take longer to come up with the expression), like transforming an enum into a switch construct for its members, or mapping said enum to a string of the member's name.
This is very far from taking over my job, though. I'd love to be more of a conductor than the guy playing all instruments in the orchestra at once.
To each their own of course. It just seems like the productivity gains are perceptual, not actual.
For an enum to a switch, I just copy the enum values and run a regex on those copied lines. Both would take me <30s, so it's a wash. That specific one would be trivial with most IDEs as well, just type "switch (variable) {" and it could autocomplete an exhaustive switch, all without LLMs.
Then again, I'm pretty old school. I still use vim as my editor (with language server plugins), and I'm really comfortable with those kinds of common tasks. I'm only going to bother learning to use the LLM if it's really going to help (e.g. automate writing good unit tests).
Sometimes those things are way more complex, for example when it's about matching over a string rather than an enum to convert it into an enum. Typing out a regex would take me maybe 10mins or more, and with the LLM I can just describe roughly what I want (since it knows the language, I don't have to explain it in detail, just something like "make this into a switch statement" is sufficient usually).
10mins at a time really adds up over a full work day.