Ah, we've got back to the "we should event source everything, and allow ourselves to change history and re-snapshot current state", idea gaining followers again.
Well, yeah. But it's hard. Also, git is not the solution to this, it's just the screwdriver you have in your hand right now.
I really really want to do version control for everything, but most of my data is binary so it really doesn't agree with git; I tried using LFS as well but it didn't work for my specific workflow.
Hopefully something less text oriented comes along in the future.
But versionning everything is not really great idea. Probably 90% of your files are static. Archives, Media, LIBs. Not worth versioning. But for that 10%, having any (D)VFS is handy. Thats why I wrote DOT for myself, to handle that niche need to version every document I work with.
why not version control those things? if they're static, there's no changes so version controlling them has little cost. maybe existing tools aren't adequate for this conception of the world, but I don't think that's a good reason not to work towards "version control everything"
It's interesting you brought up Lore. I also wonder whether importing VCS tech from the games industry for versioning general agentic work would be a good idea.
Problem with dealing with binary files with git (or source control in general) is that you want to be able to see diffs, which is very dependent on the type of binary file you have.
I think you can create git plugins to show diffs in different formats for certain file-types (or even open a 3rd party tool to let you visualize it), but it is a lot of work. Most productivity tools don't have anything of the sort.
If you just care about storing the file git LFS + manual text-changelog per binary file works well, but it is annoying to get everyone on your team to work in this workflow (heck, getting everyone and all automation scripts to install LFS is already a pain).
The approach of having everything next to the code seems good in theory but it's just a nightmare in practice for big projects.
Imagine your git history filled with commits from project managers where every little change in requirements and docs has its own commit.
We've tried that and ended up having to rebase our branches multiple times a day while we lost the overview of code changes completely.
The main issue that code can live in multiple branches while development but you can't apply that to docs and requirements where you need to have one centralized source of truth for everyone.
I think we will start to see more adaptation from collaboration tools to provide/accept the info in text format making it suitable for agents as a working solution. Meanwhile, I've seen that having two repos (One for Code and one for docs) is the best solution for the current tools available
> Design docs from Google Docs to checked-in markdown?
This worked pretty well for the team I'm in. Design docs in Google Docs were really hard to keep in sync with decisions and not properly agent accessible. Initially, we were concerned lack of comments would be an issue, but this didn't really came true - a Slack channel does a good enough job. With this flow, the design author makes a decision and review results, agent propagates it to individual areas, and from those to tasks (those are in MD too), and then copies descriptions to Jira. No more "we changed a thing but missed one place that depends on it", or at least not as bad as before.
Not the whole backlog, just the tasks for the feature being designed, like an epic. Atlassian has a CLI that - acli. We use Jira as usual, it's just a way to not having to manually propagate design adjustments to Jira.
LLMs are irremediably broken (agents also) for delivering software.
The most you can do is to accelerste subworkflows and review as hell.
Noone wants middle term maintenance nightmares or unreliable code. Can look good the first month, that's it.
I think they have been overselling it so much but you need to know exactly what you are doing and I think for generating code they are pretty bad. And they will improve as they sniff some more data but that's it. Something new, something LLMs do pretty bad.
> We use issue trackers and pull requests to manage work, people write documentation and communicate over email, instant messaging, and in meetings. Keeping all of the information in these channels synchronized and up to date is a full time job.
The crux of what OP wants seems to be transparent interfaces to the data store backing each of these services. Version control would be about tracking changes when this data mutates, which seems somewhat orthogonal to this stated need.
I wonder whether a nice solution to this would be to have a distributed architecture where each node can publish / subscribe to updates and maintain a local copy it operates on, with conflict resolution for eventual consistency.
PS: Checkout Perkeep for a take on aggregating all your data in one place.
The revert safety net is what makes agentic coding tolerable. Without version control, every AI mistake becomes permanent, and that alone kills the workflow.
Ah, we've got back to the "we should event source everything, and allow ourselves to change history and re-snapshot current state", idea gaining followers again.
Well, yeah. But it's hard. Also, git is not the solution to this, it's just the screwdriver you have in your hand right now.
I really really want to do version control for everything, but most of my data is binary so it really doesn't agree with git; I tried using LFS as well but it didn't work for my specific workflow. Hopefully something less text oriented comes along in the future.
Lore looked interesting for this purpose. https://github.com/EpicGames/lore
But versionning everything is not really great idea. Probably 90% of your files are static. Archives, Media, LIBs. Not worth versioning. But for that 10%, having any (D)VFS is handy. Thats why I wrote DOT for myself, to handle that niche need to version every document I work with.
Its private for now, but here is printout of help: http://borg.uu3.net/~borg/?dot
The bigger issue is that most of the files you're likely to capture in "version everything" are just noise.
Disk space is far from cheap when the value of what's being stored is basically zero.
You can version everything today with something like ZFS snapshoting - and very quickly realize hundreds of gigabytes of noise aren't worth keeping.
why not version control those things? if they're static, there's no changes so version controlling them has little cost. maybe existing tools aren't adequate for this conception of the world, but I don't think that's a good reason not to work towards "version control everything"
It's interesting you brought up Lore. I also wonder whether importing VCS tech from the games industry for versioning general agentic work would be a good idea.
Why not just fork git and make it better at binary files?
Problem with dealing with binary files with git (or source control in general) is that you want to be able to see diffs, which is very dependent on the type of binary file you have.
I think you can create git plugins to show diffs in different formats for certain file-types (or even open a 3rd party tool to let you visualize it), but it is a lot of work. Most productivity tools don't have anything of the sort.
If you just care about storing the file git LFS + manual text-changelog per binary file works well, but it is annoying to get everyone on your team to work in this workflow (heck, getting everyone and all automation scripts to install LFS is already a pain).
The approach of having everything next to the code seems good in theory but it's just a nightmare in practice for big projects.
Imagine your git history filled with commits from project managers where every little change in requirements and docs has its own commit. We've tried that and ended up having to rebase our branches multiple times a day while we lost the overview of code changes completely.
The main issue that code can live in multiple branches while development but you can't apply that to docs and requirements where you need to have one centralized source of truth for everyone.
I think we will start to see more adaptation from collaboration tools to provide/accept the info in text format making it suitable for agents as a working solution. Meanwhile, I've seen that having two repos (One for Code and one for docs) is the best solution for the current tools available
> Design docs from Google Docs to checked-in markdown?
This worked pretty well for the team I'm in. Design docs in Google Docs were really hard to keep in sync with decisions and not properly agent accessible. Initially, we were concerned lack of comments would be an issue, but this didn't really came true - a Slack channel does a good enough job. With this flow, the design author makes a decision and review results, agent propagates it to individual areas, and from those to tasks (those are in MD too), and then copies descriptions to Jira. No more "we changed a thing but missed one place that depends on it", or at least not as bad as before.
Would you mind elaborating on
please?We're doing the design docs in MD thing, but are you saying you've gone a step further and have the backlog represented in git too?
Not the whole backlog, just the tasks for the feature being designed, like an epic. Atlassian has a CLI that - acli. We use Jira as usual, it's just a way to not having to manually propagate design adjustments to Jira.
LLMs are irremediably broken (agents also) for delivering software.
The most you can do is to accelerste subworkflows and review as hell.
Noone wants middle term maintenance nightmares or unreliable code. Can look good the first month, that's it.
I think they have been overselling it so much but you need to know exactly what you are doing and I think for generating code they are pretty bad. And they will improve as they sniff some more data but that's it. Something new, something LLMs do pretty bad.
> We use issue trackers and pull requests to manage work, people write documentation and communicate over email, instant messaging, and in meetings. Keeping all of the information in these channels synchronized and up to date is a full time job.
The crux of what OP wants seems to be transparent interfaces to the data store backing each of these services. Version control would be about tracking changes when this data mutates, which seems somewhat orthogonal to this stated need.
I wonder whether a nice solution to this would be to have a distributed architecture where each node can publish / subscribe to updates and maintain a local copy it operates on, with conflict resolution for eventual consistency.
PS: Checkout Perkeep for a take on aggregating all your data in one place.
The revert safety net is what makes agentic coding tolerable. Without version control, every AI mistake becomes permanent, and that alone kills the workflow.
I've been working for a long time to be ready for this need! Blab is an (un-announced) version control system designed to support "everything".
It's built on top of a content-hashable XML-ish data modeling language called CSTML: https://docs.bablr.org/guides/cstml
Binary document support isn't done yet, but it is planned.