Rework log: multithreaded engine - PLEASE HELP TESTING

[mcallegari]

As anticipated in my previous rework logs, QLC+ 4.12.0 is going to be a huge release for this project.
So, here's another MASSIVE rework happened in the QLC+ code base.

After continuous reports of crossfading cases not working, I decided to face one major change I postponed for too many years.

This rework aims to tackle two major issues of the QLC+ engine:
- Universe composition chain
- fade order not covering all usage cases

The task has been tracked on GitHub, and there you can find also a document with block diagrams explaining the changes.

The QLC+ engine was single threaded. Meaning, everything happened in a single loop, totally ignoring your CPU ability to spawn tasks across cores.
Ironically, a single 3GHz core was better than four 1Ghz cores.

With this rework, instead, Universes are treated as independent tasks (threads) and your OS will decide how to spawn them across the available physical/logical cores.
Plus, each Universe "final composition" is now triggered by a single event, therefore they work in parallel, and not in sequence like it was before.
This gives more breath to the main QLC+ engine clock (aka Master Timer) during its 20ms period, since it runs at 50 Hertz.
It "only" needs to gather from Functions, VC Widgets and Simple Desk, the faders involved and program them on their related Universe.
This is done just once, unless a fader changes continuously. All the rest (fade in, fade out, blending) is handled by Universes.

You might understand this is a very different approach than before, so I really need all the possible help to test this big change.
Of course I already carried out a large number of tests, either manual or automatic. From what I saw, the current state looks pretty good.

This rework includes also a review of part of the Chaser logic. The good theatre people suffered from non functional cases when using the VC Cue list widget and side faders.
So I've had a closer look at it as well and implemented a way to start Chaser steps with all the information they need to behave as expected. (e.g. startup itensity, blend mode)

So, once again, please help testing! I've updated the test versions for all platforms. Pick one, open your projects and check (at least with the DMX/2D monitor) that everything works like before.
The more we test, the better QLC+ 4.12.0 will be. Please spare me the need to rebuild everything one week after I release, because it costs me a lot of time and effort.

I plan to release by the end of the month, which is actually the planned 6 months release cycle for QLC+ 4.
So, there's plenty of time to make this 4.12.0 a great release!

Thanks in advance to everyone who can dedicate some time to testing.

[edogawa]

Massimo, this is great news and I'll test this intensively during the next few days (I'm one of the good theater people). Our next opera project will take place in November, and I hope to be able to run this new version there.

I was running a show end of September, using a very recent 4.12 from git and had minor problems with cuelist playback. Especially with differences between manual and timed crossfades, which seem to use very different code paths from what I can understand from reading the code. For example, Intensity of a certain channel would change during a manual crossfade, but not during a timed "Go button" playback, if that same channel was active from a scene button playback. Nothing that I wasn't able to work around, but If any these issues remain, I'll collect them in a dedicated bug report.

I'd just like to point out once more that the openSUSE Tumbleweed OBS build is broken for all your package variants and still has binaries from 2017. Easiest solution seems to be to include an older version of google protobuf in home:mcallegari79 and build ola against that. I've done that in my branch of qlcplus-qt5 if you (or cingulingu) want to have a look. The most recent ola release should fix this IIRC, but seemingly this still doesn't work...

But please support Tumbleweed, I think although it's "rolling release", nowadays with all that QA in place it is stable enough to be used in production...

Edgar

[mcallegari]

I'd just like to point out once more that the openSUSE Tumbleweed OBS build is broken for all your package variants and still has binaries from 2017. Easiest solution seems to be to include an older version of google protobuf in home:mcallegari79 and build ola against that. I've done that in my branch of qlcplus-qt5 if you (or cingulingu) want to have a look. The most recent ola release should fix this IIRC, but seemingly this still doesn't work...

Yes, I'm aware of broken releases and the idea was waiting for OLA 0.11.0 to be released.
The fix has been merged on February 19th, but still no sign of an OLA release including it.
Apparently they still have 25 bugs to fix before 0.11.0 can happen.
In other words, I'll try to fix the OBS build myself as you suggested. I just need to find some time among all the things ongoing in QLC+.

[mlohrey]

Hi Massimo,

I have done some testing of a simple cue list and attached the work space.

There is some strange unexpected behaviours in a few places.

See attached workspace.
Click each button to show the expected result for 4 basic scenes and simple sequence. This all works as normal

Now hit play on the cue list and step through the cue list. All is good until you reach the sequence. This is probably just the old issue of fade times in the cue list being applied to the sequence. Ultimately, this sequence should fade up to the full 'bounce' effect.

Start at the first cue in the cuelist. Now, link the side faders, and move them up slowly. All is good. Move them right to the top. Still good. Move very slowly down and you will notice that the cross fade jumps violently to zero. This is unexpected.

Tested on MacOS 10.14 Mojave

Cheers

Mark

Bounce Test.qxw

(13.47 KiB) Downloaded 394 times

[mcallegari]

Hi Mark, thanks for sharing a sample workspace.
I see what you mean, and I'll try to find a solution to that.
I think at the moment the override fade influences the fade of the sequence steps, which is not what you want.
A Chaser/Sequence as a step of a Chaser should definitely be treated as a special case.
Or the other way around: Scenes should be the special case, and all the other functions (e.g. Collections, EFX, etc) should be subject to a different fade logic.

[edogawa]

Hello again,

I've just compiled and installed current git master locally, I hope that's ok to test the changes?

I'm testing using my latest opera show file, which I cannot test with real dimmers and LED fixtures ATM, as we've already striked the show.
But from looking at the DMX monitor, I see some glitches, and even if I'm quite sure that some have been there before this mtengine merge, I'll try to describe them here.

The cuelist consists of scenes only, with in/out fade times set individually in the chaser editor, as usual.
Some helper functions are referenced from the buttons labelled Dirigent, Orchester and Technik. These are halogen lamps set to certain levels that ought not to change during the whole show until final blackout. I used them during rehearsal and while programming / editing the scenes. There's another helper function that sets all the dimmers (except for channel 16 but that's ok as it was unused, ignore that) to DMX #55/27% as a preheat value, because the dimmers I was using there are incredible crap and indeed starting to output only around 30%. Due to my problem described here viewtopic.php?f=20&t=10774#p53443 (which only shows in KDE as it now turned out) I had to compensate this way instead of using channel modifiers, to achieve acceptable dimming curves/fades.

1.
Now if you start my cuelist, fades behave very differently depending on using the next cue button or the (linked) manual crossfader. Also when clicking a scene entry that's earlier in the list than the current cue, or the previous cue button, strange things happen. the preheat values change in most cases except when going in normal order by pressing "next cue" for a timed "Go". When using the crossfader, the preheat drops to different values depending on I don't know what, maybe fade times? When going backwards by button or by clicking a list item directly, the preheat values go higher. Well I actually have difficulties seeing a pattern when changing cues randomly by the methods described.
I remember seeing similar effects , and that was the main reason to add fade times and run the show by "next cue" keypress instead of manual crossfade, which I normally prefer. (the second reason being flickering LED lights when using nanoKontrol2 faders or knobs for crossfading, either they are THAT crappy or the 7 bit resolution just doesn't cut it... halogen lamps are slow enough to hide that, but LED light shows that brutally unless the light has some internal DMX fade speed parameter)

2. When one of my helper scene buttons is active during cuelist playback, the associated channels behave wrongly, for example switch on "Technik" and you'll see Dimmer channel 12 jumping from the programmed value 180 to 255 and also changing the actual value during fade, this also occurs while manually fading. This issue was already present before the merge.

3. Occasionally, and this is definitely new since the merge, when I mouse-drag the (linked) right crossfader from up to down rather quickly, the next cue seems to get triggered when I arrive at the bottom end.

These are my main concerns at this point. I'll be in that same venue again during the second half of November, and hope we can iron out some of them until then

Attached are my workspace and custom fixture definitions. I'm running out of time now, and as i'm not sure if any of the fixtures already have been submitted/included in QLC+, I thought I better include them here.

[edogawa]

I don't seem to be able to add a fourth attachment with my workspace to my previous post, so here it comes.

[floEdelmann]

I've tested with multiple Art-Net universes and found no issues

[mcallegari]

I don't seem to be able to add a fourth attachment with my workspace to my previous post, so here it comes.

[moderator hat on] perhaps attach a single ZIP file next time ?

I just pushed a change that makes VC Slider playback of a Sequence behave correctly. This was a long standing issue that's always been there.
Having Mark's Sequence fading for 6 seconds as a step of a Chaser is another story... And honestly, the more I think about it, the more I am confused about what the expected result can possibly be.
You want to go from an infinite Scene step with values: 0, 255, 0, 255, 0
to a Sequence with 400ms steps that are supposed to fade in for 6 seconds considering the values above.
I think my head is going to explode if I try to find a solution to that.

So, please, one step at the time. Can you guys please tell me if currently there are regressions compared to 4.11.2 ? If so, test project please. The simpler, the better.
@Edgar: TLDR let's keep it simple for now. You're using channel modifiers that make analyzing whatever you described even more difficult.

Once all regressions are fixed, then we can start analyzing more complex scenarios.

[edogawa]

Sorry, I didn't think of zipping my files...

I apologize for mixing things up and mixing testing this special branch merge with general bugreports.

I got aware and felt bad for this right after sending my post... and I'm still not sure if all this belongs in this thread or elsewhere. I think I observed all these issues before your merge of mtengine branch. The thing described as "definitely new since the merge" in my post from above might have been me not understanding first what I describe below (the progress bar still moving after manual crossfade has reached the target cue). I apologize again.

@Edgar: TLDR let's keep it simple for now. You're using channel modifiers that make analyzing whatever you described even more difficult.

No, there are no channel modifiers involved, as I wasn't able to engage them in my project, as I wrote in the other thread.

Instead, what I've done was to simulate a preheat modifier preset by adding a base level of DMX #55 (27% intensity) to all my "conventional" dimmer channels in all my show scenes. (I seem to have overlooked one channel that wasn't in use, but that's not relevant).

I'm sorry for this TLDR but this has again become a lengthy post, as I find it hard to describe my problems in a reproducible way. Given that this is the wrong thread anyway I think we should re-open that old discussion on traditional theater use cases / hardware lightboard philosophies / cuelist design and try to solve this finally... not sure if here in the forums or a github issue/milestone, anyway I think it needs to be solved. I can try to write up the functioning, pros and cons of the few I know well, i.e. Strand LBX/GSX and 430/530 families and Compulite Spark, from the 1990s ... if that would help...

I tried now to set up a simple workspace to demostrate my issue like Mark did. I hope my issue becomes clear now, and I think it's related to Mark's, it also shows sudden level jumps.

- Two dimmer channels, representing halogen lights.
- Two buttons with scenes for each of these at 50%
- Two faders with override/monitor activated, one for each dimmer channel
- a cuelist with six simple scenes (no sequence or other functions): all off, 1@50, 1 + 2 @ 50, 1@F, 2@50, 2@F, Xfade visible, in Blend mode and linked. fade times set to per-step but all steps are at 3 seconds fade in/out for demonstration.

Switching a button, moving any channel fader or running the cuelist by "next/previous cue" buttons: Each of these actions taken individually and without the others, works as expected.

But, for example:
- switch on button "1 @ 50 %", from step 0 hit "next cue" button again: fade to step 2 behaves correctly.
- press "next cue" again, channel 1 during this fade increases to 100% (incorrect, it should stay at 50%)

Also when I use the cuelist crossfade, several unexpected things happen:

1. If I grab the fader that shows 0%, and rapidly move it to say 1/3 of the fader length, the corresponding crossfade happens but not immediately, instead with the programmed fade times. Grabbing the fader that shows 100%, crossfade follows immediately.

2. if any of the scene buttons are active, I see this:

- start the cuelist at step 0, all is well
- grab any crossfader and move it a bit, channel 1 drops to 0% and continues to follow the fade according to the cuelist, ignoring the button value that should have priority in HTP. The observation from 1. above still applies. I find it hard to describe precisely, but if you play a bit you'll sort of see a pattern. Sometimes when you fade quickly to next cue, the intensities follow immediately but the progress bar still shows running after having reached the target cue.
-being in cuelist step 2 (scene "1 @ 50%" again), deactivating the same scene from the button matrix brings that channel down to zero. I'd expect it to stay at 50% like the cuelist dictates, same HTP rule.

So to achieve a dipless crossfade from scene to scene, I currently must use pre-programmed fade times and next/previous cue button "go" functionality, and at the same time make sure that no scene button is active if it includes channels that also exist in the cuelist...

And now that I write this, I remember from the light rehearsals that the same happened with Pan/Tilt of my two small moving washlights: I programmed them into all the cues with explicit Pan/Tilt values, with occasional small position changes only. When running these cues the Pan/Tilt would jump to zero with each crossfader move but also when using "next cue" button, returning to the same static position at the end of a fade. I had to deactivate the Pan/Tilt channels throughout the cuelist scenes except where a move was wanted. Same for Zoom on the LED Zoom PARs.

[mlohrey]

Hi Massimo,

I have tested 4.12.0 GIT with as many of my old show as I could find and couldn't find any issues that weren't there in 4.11.2

I don't want your head to explode!

I noticed this behaviour in a production I did ,where a scene started with a dance with flashing lights. I wanted to bring the scene up slowly rather just start with full intensity. In the end, I think I might have just used the GM slider and did it manually, as the previous scene was a blackout anyway.
A while back you shared this document (it is still a well above my head!)
I really have no idea how you program fades and I am not sure whether the document is still current but it seems to me there is a need for fade hierarchy similar that of the various elements and that cuelists fades apply to the whole sequence, rather than the individual steps a bit like the master control does.
I am happy to model the fade in my simple example in an iterative way if that helps. Let me know.

Thanks for your work on this.

I will go back to testing and leave you worry about this some other time.

Cheers

Mark

[edogawa]

Please use the URL tag, or something else went wrong, that link is crippled...

[mcallegari]

I fixed the URL. However, that's the "old" architecture.
As pointed out in the OP of this thread, the new architecture diagram is this.

The idea is that every QLC+ component contributes with faders in the final universe composition, with the right order.
LTP channels always overwrites the final value, while HTP channels follow the HTP rule.
Fades are just a "from - to" timed rule. The tricky part is handling nested requirements like a Chaser into a Chaser. Handling timing overrides is not easy at all, even with the new architecture.

@Mark: are you building from sources ? Cause when I wrote my last post the macOS test build was not updated. Now it is.

[mlohrey]

Thanks for updating the test build. I have a new Mac and was just deciding whether I should update all the compiling tools and do some building. I will download and test.

[maximortal]

PC w10
qlc+ build 4.12.112

workspace attached

some bug that I found

1: in function editor if you try to use prewiew feature (cheasers) the DMX monnitor show nothing. no idea if this is also in "real world" or just a visualization bug

2: if you assign a chaser to a slider in virtual console if you move the slider the chaser is played as expected, soon you let in place the slider ( no matter if at fulll or any other level) the chaser is not reproduced at all

3: as before build ( so this is not a regression ) chaser nested inside another chaser ignore the fade in and out of nested chaser

N.B.: this is just a suggestion for Massimo: if a user want to use qlc+ for a theater show must be strongly warned that if he don't enable the MIX mode in cue list widget THEATRICAL programming mode do not work properly (I can explain that better if requested)

[mcallegari]

Hello Iro,
#1 should be fixed now. Please check the latest test build.
#2 I don't understand what you mean. Can you please elaborate ?

[maximortal]

4.12.122

#1 fixed
#2 fixed too so...it doesn't matter anymore

[mcallegari]

4.12.122
#1 fixed
#2 fixed too so...it doesn't matter anymore

Yay !

[mcallegari]

@Edgar: thanks to a patch provided by the OLA guys, all qt5-git OBS builds are now successful, including Tumbleweed.

[edogawa]

Oh cool, thanks for letting me know!