63
submitted 5 months ago* (last edited 5 months ago) by matcha_addict@lemy.lol to c/linux@lemmy.ml

Hi all,

I am looking for a local database that is easily accessible via the command line.

It can be SQL or non-SQL

Whats my use case? I want to use it kinda like a second brain. A place to save ~~my notes~~, my todo lists, my book reading lists, links / articles to read later, etc.

I want it to be a good CLI citizen so that I can script its commands to create simpler abstractions, rather than writing out the full queries every time.

Maybe sqlite is what I need, but is that ideal for my use case?

Edit: removed notes, as evidently they aren't suitable for this and aren't like the rest.

top 49 comments
sorted by: hot top controversial new old
[-] AnnaFrankfurter@lemmy.ml 36 points 5 months ago

I wouldn't recommend a DB for note taking purpose. You can use Markdown with vim.

[-] fmstrat@lemmy.nowsci.com 2 points 5 months ago

Agreed. You can still do tags with filenames like ===Category===Tag 1---Tag 2===Note name.md or some such format.

[-] matcha_addict@lemy.lol 1 points 5 months ago

Perhaps I shouldn't have said note taking. But the rest still applies.

[-] Emperor@feddit.uk 29 points 5 months ago

Counter-proposal: Perhaps use Obsidian instead?

[-] Brewchin@lemmy.world 5 points 5 months ago

I second this, as it's my use case.

Providing you lay out each note correctly with appropriate frontmatter, Dataview's DQL and DataviewJS give you all the SQL-like functionality you could want.

Plus a load of useful functionality beyond a plain DB.

[-] asap@lemmy.world 6 points 5 months ago* (last edited 4 months ago)

Add in the local REST API as well if you want to easily interact with your notes programmatically:

https://github.com/coddingtonbear/obsidian-local-rest-api

[-] Brewchin@lemmy.world 5 points 5 months ago

Had no idea about this. Very useful, thanks!

[-] matcha_addict@lemy.lol 1 points 5 months ago

I want something I can use from the command line, so that I can integrate and interface it with other unix tools and scripts.

[-] Emperor@feddit.uk 3 points 5 months ago
[-] matcha_addict@lemy.lol 1 points 5 months ago

Oh wow that's pretty cool. From my understanding this is good for notes, but would it work for tasks / todos?

[-] Emperor@feddit.uk 1 points 5 months ago

It has a huge number of plugins - this is a list of the productivity ones. You'd need to do some reading to see if it has what you require.

[-] cy_narrator@discuss.tchncs.de 14 points 5 months ago

Using a database for notes is like going 2km in a plane.

You can use any relational database for this though but why would you subject yourself to this?

[-] matcha_addict@lemy.lol 2 points 5 months ago

I think you're right about notes, but what about the other use cases?

[-] cy_narrator@discuss.tchncs.de 1 points 3 months ago

If you are generally interested in designing databases its a good exercise to try and build one yourself. That is the only situation I recommend you go this route

[-] Emperor@feddit.uk 1 points 5 months ago

Obsidian has a huge number of plug-ins, that cover a wide range of uses. I also use Zettel Notes as a quick md file editor and that has the ability to capture pages in markdown to read later. And/or you can use Omnivore.

[-] OneCardboardBox@lemmy.sdf.org 14 points 5 months ago* (last edited 5 months ago)

Are you an emacs user?

Try org-roam. It's a similar system to obsidian, but fully open source. You have all the note taking techniques of org-mode, and all the scripting power of emacs.

[-] sem@lemmy.ml 3 points 5 months ago

There is also a "web-version" of org-roam named "orgnote": https://github.com/Artawower/orgnote

[-] pe1uca@lemmy.pe1uca.dev 10 points 5 months ago

I can't imagine this flow working with any DB without an UI to manage it.
How are you going to store all that in an easy yet flexible way to handle all with SQL?

A table for notes?
What fields would it have? Probably just a text field.
Creating it is simple: insert "initial note"... How are you going to update it? A simple update to the ID won't work since you'll be replacing all the content, you'd need to query the note, copy it to a text editor and then copy it back to a query (don't forget to escape it).
Then probably you want to know which is your oldest note, so you need to include created_at and updated_at fields.
Maybe a title per note is a nice addition, so a new field to add title.

What about the todo lists? Will they be stored in the same notes table?
If so, then the same problem, how are you going to update them? Include new items, mark items as done, remove them, reorder them.
Maybe a dedicated table, well, two tables, list metadata and list items.
In metadata almost the same fields as notes, but description instead of text. The list items will have status and text.

Maybe you can reuse the todo tables for your book list and links/articles to read.

so that I can script its commands to create simpler abstractions, rather than writing out the full queries every time.

This already exists, several note taking apps which wrap around either the filesystem or a DB so you only have to worry about writing your ideas into them.
I'd suggest to not reinvent the wheel unless nothing satisfies you.

What are the pros of using a DB directly for your use case?
What are the cons of using a note taking app which will provide a text editor?

If you really really want to use a DB maybe look into https://github.com/zadam/trilium
It uses sqlite to store the notes, so maybe you can check the code and get an idea if it's complicated or not for you to manually replicate all of that.
If not, I'd also recommend obsidian, it stores the notes in md files, so you can open them with any software you want and they'll have a standard syntax.

[-] matcha_addict@lemy.lol 1 points 5 months ago

First, I want to apologize for mentioning notes. Notes should be separate from the rest, and I agree with you on what you said about notes.

For the rest, I've asked before on suggestions for apps to save my todos, links, etc. There were a couple issues

  • I couldn't find something that gives me the same interface for all of these. They are all lists of things with some attributes and relations, but most apps out there handled each one of these separately and differently

  • they lacked features I wanted. For example, dependent tasks, a flexible tag system I can query by, etc.

A DB would let me do all of that.

[-] pe1uca@lemmy.pe1uca.dev 1 points 5 months ago

Ah, that makes sense!
Yes, a DB would let you build this. But the point is in the word "build", you need to think about what is needed, in which format, how to properly make all the relationships to have data consistency and flexibility, etc.
For example, you might implement the tags as a text field, then we still have the same issue about addition, removal, and reorder. One fix could be have a many tags to one task table. Then we have the problem of mistyping a tag, you might want to add TODO but you forgot you have it as todo, which might not be a problem if the field is case insensitive, but what about to-do?
So there are still a lot of stuff you might oversight which will come up to sidetrack you from creating and doing your tasks even if you abstract all of this into a script.

Specifically for todo list I selfhost https://vikunja.io/
It has OAS so you can easily generate a library for any language for you to create a CLI.
Each task has a lot of attributes, including the ones you want: relation between tasks, labels, due date, assignee.

Maybe you can have a project for your book list, but it might be overkill.

For links and articles to read I'd say a simple bookmark software could be enough, even the ones in your browser.
If you want to go a bit beyond that I'm using https://github.com/goniszewski/grimoire
I like it because it has nested categories plus tags, most other bookmark projects only have simple categories or only tags.
It also has a basic API but is enough for most use cases.
Other option could be an RSS reader if you want to get all articles from a site. I'm using https://github.com/FreshRSS/FreshRSS which has the option to retrieve data form sites using XMLPath in case they don't offer RSS.


If you still want to go the DB route, then as others have mentioned, since it'll be local and single user, sqlite is the best option.
I'd still encourage you to use any existing project, and if it's open source you can easily contribute the code you'd have done for you to help improve it for the next person with your exact needs.

(Just paid attention to your username :P
I also love matcha, not an addict tho haha)

[-] tmk@social.lugal.io 9 points 5 months ago

@matcha_addict Anything beyond SQLite is too heavy, but if it were me I would use a lightweight wiki like dokuwiki. Having to run SQL to do all these sorts of things sounds like it would just get in the way of both getting and consuming your thoughts.

[-] matcha_addict@lemy.lol 1 points 5 months ago

I don't think it would suffice. It would work for notes, but I want to also have a todo list in there, for example. Be able to check things off, query by due date, by assignee or task, etc.

[-] subignition@fedia.io 13 points 5 months ago

I don't think a database is a good fit for your use case at all, unless you're willing to reinvent a lot of wheels.

[-] matcha_addict@lemy.lol 2 points 5 months ago

What would then? I haven't found anything that has command line interface and supports dependent tasks and a flexible, queryable tag system

[-] subignition@fedia.io 1 points 5 months ago

I'm sorry, I don't have a suggestion. If it were me I would not be looking at a CLI solution for this at all.

[-] atzanteol@sh.itjust.works 4 points 5 months ago

For todo stuff there's the excellent "todo-txt" for cli use. Though not great for notes. Perhaps separate tools would be fine?

[-] matcha_addict@lemy.lol 1 points 5 months ago

Todo txt lacked some features I needed, which would be trivial to implement as a SQL column.

I do not remember what it was... Maybe it was dependent tasks? I believe so.

[-] makussu@feddit.org 2 points 5 months ago

Org mode, org roam, org agenda and org-ql

[-] Daeraxa@lemmy.ml 8 points 5 months ago

Joplin is a note taking app that stores its data in an sqlite database (easy to query but not a good idea to write to it) but there is also a command line version and both versions support access via a data API.

[-] delirious_owl@discuss.online 7 points 5 months ago

Is their any DB that doesn't have a CLI?

[-] matcha_addict@lemy.lol 1 points 5 months ago

I think what I'm looking for doesn't exist, but what I meant by CLI is something I can pipe things into and interface with other unix tools easily.

But you're right, they all have a way to open a session via CLI.

[-] terminal@lemmy.ml 7 points 5 months ago

I think you would be better off using something like org-roam. It’s all text so script can still be used and it can be searched fast with ripgrep. Also org mode has loads of features that a homegrown system will never be able to catch up with

[-] bionicjoey@lemmy.ca 5 points 5 months ago

Definitely sounds like sqlite.

[-] Nibodhika@lemmy.world 5 points 5 months ago

What if instead you used something that's meant to be used to take notes but that also has querying capabilities?

I have been using Silverbullet for a while and I absolutely love it. It uses Markdown files in disk so it's very easy to backup, have secondary instances running and even just edit files directly with any other program. But also provides some extra syntax to define objects and query them, so you can for example build a library of recipes and have a page that lists all of the ones that have a specific tag or take less than X time to cook or whatever.

[-] mvirts@lemmy.world 5 points 4 months ago
[-] csm10495@sh.itjust.works 5 points 5 months ago

Unless your lists are 1000s of items long, just use text files in a folder.

Could even go fancy and use markdown.

[-] matcha_addict@lemy.lol 1 points 5 months ago

I definitely considered this approach. The only problem is it is difficult to represent linking items to each other. For example, dependent tasks or sub tasks.

[-] h0bbl3s@lemmy.world 4 points 5 months ago* (last edited 5 months ago)

They aren't exactly CLI but I really like obsidian for taking notes. It's not open source though. Logseq is good too and is OSS. Both use markdown for formatting so if you are familiar with writing pages on GitHub you'll have no trouble. Even if not markdown is super easy to learn. That and all of your data stays local and in open formats. I edit my stuff in a terminal anyway.

Just look up obsidian OSINT on YouTube you'll find some good stuff on how to use it.

Another thought is just use markdown files and a directory structure in a private git repo. You'd be able to interact with it locally entirely in the terminal with vim etc and have the option of going online and searching or organizing etc. You could probably even use a cli browser for that part if you wanted.

[-] audaxdreik@pawb.social 4 points 5 months ago

Check out https://www.giuspen.net/cherrytree/, lightweight note-taking app with interesting scripting function built in.

Even if that's not your cup of tea, it has the option to save your notebook to a single sqlite file, so I take that as good enough proof it'll work for your similar purposes as well.

[-] elltee@lemmy.one 3 points 5 months ago

Joplin can do all that and has: Plugin support for nearly anything you want Runs on sqlite if you want to access the data direct Is 100% open source.

[-] lazylion_ca@lemmy.ca 1 points 5 months ago

Also has (or had) a cli interface.

[-] golden_zealot@lemmy.ml 1 points 5 months ago

I didn't know this about Joplin, that's pretty neat.

[-] gramgan@lemmy.ml 3 points 5 months ago

Hey!

I’ve wanted something very similar—specifically, a plain-text database. I recently came across GNU recutils, which I haven’t had time to play around with yet, but which seems like it fits the bill (at least for me). There’s a couple YouTube videos on it—I encourage you to check it out!

[-] floofloof@lemmy.ca 2 points 5 months ago

If you want something a bit more powerful than SQLite, MySQL and PostgreSQL both support CLI interactions and scripting too.

[-] olafurp@lemmy.world 2 points 5 months ago* (last edited 5 months ago)

You can always log into a database and use raw SQL. For automation of your tasks you can put in some functions and stored procedures and later on you can always do something more graphical or even more SQL. You can do something like "exec create_note('Name of note')" and "exec notes()" and whatnot.

Then if you want to do something more elaborate than viewing tables and do some formatting programming languages are pretty nice do make stringified versions of the notes and a nice way to browse them.

Then to finish up using some nice bash packages to create a CLI interface is very fun. I like charmbracelet's stuff because it just looks nice and is easy to use. Here's a thing you can use to create something to select a note to view: https://github.com/charmbracelet/gum

[-] yogthos@lemmy.ml 1 points 5 months ago* (last edited 5 months ago)

take a look at Apache Ignite https://ignite.apache.org/docs/latest/tools/sqlline

there's also datafusion that lets you run SQL commands via CIL on CSV and JSON https://datafusion.apache.org/user-guide/cli/usage.html

[-] vipaal@aussie.zone 1 points 5 months ago

Tangential answer. Consider looking into Prolog, Picat, Mercury languages. You can effectively let the database design be taken care of by the language. In return you get more time to reflect on your knowledge base and ask it all sorts of questions and get a range of possible answers.

Org-roam and its web cousin webnotes both have solved designing the database for note taking purpose using g sqlite as a back end. Good options.

[-] k4j8@lemmy.world 1 points 5 months ago

No command line interface, but if you're focus is a single solution with a consistent interface for lists, to-dos, etc., AppFlowy might be what you are looking for.

I'm a huge fan of NocoDB, including their kanban views, group by options, and forms. You could use the GUI to create the tables and relations and then use the REST API to quickly update from the command line. It can use any database for its storage, so you could still create scripts or read the data for specific needs.

this post was submitted on 23 Jul 2024
63 points (100.0% liked)

Linux

48658 readers
402 users here now

From Wikipedia, the free encyclopedia

Linux is a family of open source Unix-like operating systems based on the Linux kernel, an operating system kernel first released on September 17, 1991 by Linus Torvalds. Linux is typically packaged in a Linux distribution (or distro for short).

Distributions include the Linux kernel and supporting system software and libraries, many of which are provided by the GNU Project. Many Linux distributions use the word "Linux" in their name, but the Free Software Foundation uses the name GNU/Linux to emphasize the importance of GNU software, causing some controversy.

Rules

Related Communities

Community icon by Alpár-Etele Méder, licensed under CC BY 3.0

founded 5 years ago
MODERATORS