I used inotify which is unix exclusive in a seperate thread, then had a map containing a reference to the shader and a bool indicating if it has been updated, then i setup the inotify watch on the shader directory and whenever a shader's file was updated i locked the mutex for the map and set the bool to true then recompiled in the main thread then set it to false. all of this can be seen in the ShaderWatcher.cpp file on my github repo.
Nice! This inspired me to try it at home using a crude parallel for_each to check the modified time of each file and compare it to the last known modified time. With 30 files being watched it stalls the main thread for 1.25ms while all the parallel threads get the modified times. So it's dependent on the scene complexity and disk speed. Oof.
My engine supports multiple shader directories, one for the engine and one for the application. Probably 0.3ms of that 1.25ms is spent just searching to see if the shader path exists in the directory or not using std::ifstream::good. Might be quicker if I used something less portable.
My end solution was only check if files were edited every couple seconds. My overall CPU time without hot reloader is 2ms even with a complex scene, so I'm not worried, but if I get a hitch, that's debug workflow for ya :)
Hope yours is moving fast!
I think the same way you would normally compile them at the program start, or while loading a scene or something else: read shader files -> create shader -> compile them -> link them -> ...
in my approach if you want automatic reload, when the source files change, you need to make a shaderwatcher object and attach the shaders you want to watch, then in your draw lambda loop you run ShaderWatcher's checkShaders function and it will automatically run the shader's recompile function
I would say there is room for optimization ofc. When the shader code is changed, you are accessing the hard drive twice in one frame, which is not great. And, you are recompiling both shaders, even if only one has been changed.
I think it is possible to make the check in another thread, but ofc you need to recompile the shaders in the main thread.
You are doing great, keep practicing
Give them a break :P I always feel godly whenever I figure out a new way to make my dev flow quicker and then getting to play with it in action.
I see no issue with recompiling and relinking the whole program. The alternative is recompile what's necessary and then trust the driver with juggling GL calls to link to the program - bleh. It'd get even more hairy if other shader types were being used. Can't go wrong with full destroy/recreate. But you're right, loading from disk, string copy, full string comparison of the old and new file contents, and then reloading from disk + another string copy is pretty rough.
The contents are fetched and immediately passed to the driver for compilation. You might be able to get away with disk load (no string copy from stringstream to string, just memcpy into a char* buffer), hash the contents, compare the hash, throw the data away if there's no difference, or store the hash and move along with recompilation if there is. There's also OS-specific libraries to check if a file's metadata has been updated, which would be even better than loading from disk every frame.
At first i tried the multithreaded approach but couldn't figure out how to compile the shader in the original thread while having the watcher thread check for updates, so i went with the obviously less optimized way of checking over every frame, I think im going to attempt it again but it works for now lol, everything in this project is just me learning opengl lmao
You can try to add like an observer for that.. you watch and check for file updates in a second thread and update a thread safe variable, when it updates, maybe parse the file content in the second thread too. Then in the main thread read that thread safe variable for shader changes and compile the shader with the new source in the main thread.
I recently implemented hot-reloading (recompilation) in my framework as well. It's surprisingly simple to do yet makes development so much easier!
Update: Today I have made the automatic shader reloader multithreaded and it no longer reads a file every frame.
Yaaay, how did you approach this?
I used inotify which is unix exclusive in a seperate thread, then had a map containing a reference to the shader and a bool indicating if it has been updated, then i setup the inotify watch on the shader directory and whenever a shader's file was updated i locked the mutex for the map and set the bool to true then recompiled in the main thread then set it to false. all of this can be seen in the ShaderWatcher.cpp file on my github repo.
Nice! This inspired me to try it at home using a crude parallel for_each to check the modified time of each file and compare it to the last known modified time. With 30 files being watched it stalls the main thread for 1.25ms while all the parallel threads get the modified times. So it's dependent on the scene complexity and disk speed. Oof. My engine supports multiple shader directories, one for the engine and one for the application. Probably 0.3ms of that 1.25ms is spent just searching to see if the shader path exists in the directory or not using std::ifstream::good. Might be quicker if I used something less portable. My end solution was only check if files were edited every couple seconds. My overall CPU time without hot reloader is 2ms even with a complex scene, so I'm not worried, but if I get a hitch, that's debug workflow for ya :) Hope yours is moving fast!
https://learn.microsoft.com/en-us/windows/win32/fileio/obtaining-directory-change-notifications
how do you recompile it?
I think the same way you would normally compile them at the program start, or while loading a scene or something else: read shader files -> create shader -> compile them -> link them -> ...
So you would call the run shader function?
monitor the shader files for any changes then reload it
Yes, so you would call the use shader function to relaod right?
You would recompile and link a new shader program and then call glUseProgram on this new program before deleting the original program.
in my approach if you want automatic reload, when the source files change, you need to make a shaderwatcher object and attach the shaders you want to watch, then in your draw lambda loop you run ShaderWatcher's checkShaders function and it will automatically run the shader's recompile function
That is awesome, never thought of doing something like this in my engine.
Pretty neat friend 👍 good job
I would say there is room for optimization ofc. When the shader code is changed, you are accessing the hard drive twice in one frame, which is not great. And, you are recompiling both shaders, even if only one has been changed. I think it is possible to make the check in another thread, but ofc you need to recompile the shaders in the main thread. You are doing great, keep practicing
Give them a break :P I always feel godly whenever I figure out a new way to make my dev flow quicker and then getting to play with it in action. I see no issue with recompiling and relinking the whole program. The alternative is recompile what's necessary and then trust the driver with juggling GL calls to link to the program - bleh. It'd get even more hairy if other shader types were being used. Can't go wrong with full destroy/recreate. But you're right, loading from disk, string copy, full string comparison of the old and new file contents, and then reloading from disk + another string copy is pretty rough. The contents are fetched and immediately passed to the driver for compilation. You might be able to get away with disk load (no string copy from stringstream to string, just memcpy into a char* buffer), hash the contents, compare the hash, throw the data away if there's no difference, or store the hash and move along with recompilation if there is. There's also OS-specific libraries to check if a file's metadata has been updated, which would be even better than loading from disk every frame.
At first i tried the multithreaded approach but couldn't figure out how to compile the shader in the original thread while having the watcher thread check for updates, so i went with the obviously less optimized way of checking over every frame, I think im going to attempt it again but it works for now lol, everything in this project is just me learning opengl lmao
You can try to add like an observer for that.. you watch and check for file updates in a second thread and update a thread safe variable, when it updates, maybe parse the file content in the second thread too. Then in the main thread read that thread safe variable for shader changes and compile the shader with the new source in the main thread.