| *** render has quit IRC | 00:04 |
| *** cgTobi sets mode: +vv Pseudonym WayneZacMP | 00:09 |
| Pseudonym | G'day. | 00:09 |
|---|
| WayneZacMP | g-day | 00:40 |
|---|
| *** cgTobi has quit IRC | 01:06 |
| *** AlexK has quit IRC | 01:06 |
| *** WayneZacMP has quit IRC | 04:10 |
| *** WayneZacMP has joined #aqsis | 04:47 |
| *** Pseudonym has quit IRC | 05:01 |
| *** Pseudonym has joined #Aqsis | 05:23 |
| *** AlexK has joined #aqsis | 05:36 |
| *** scorp007 has joined #aqsis | 06:43 |
| *** scorp007 has quit IRC | 06:55 |
| *** AlexK has quit IRC | 07:05 |
| *** pgregory has joined #aqsis | 08:04 |
| *** ChanServ sets mode: +o pgregory | 08:04 |
| pgregory | morning all | 08:05 |
|---|
| *** AlexK has joined #aqsis | 08:06 |
| Pseudonym | G'day. | 08:06 |
|---|
| *** pgregory sets mode: +vv Pseudonym WayneZacMP | 08:06 |
| pgregory | hi Andrew, how's things? | 08:06 |
|---|
| Pseudonym | OK. You? | 08:06 |
|---|
| pgregory | yeah, not bad, not bad at all | 08:06 |
|---|
| pgregory | got the FB back on track | 08:06 |
|---|
| Pseudonym | Cool. | 08:06 |
|---|
| pgregory | need to think a bit about mutex's and such | 08:08 |
|---|
| pgregory | but it's basically working, and threaded, which is nice. | 08:08 |
|---|
| * Pseudonym nods | 08:08 |
| Pseudonym | Excellent. | 08:08 |
|---|
| *** Anteru has joined #aqsis | 08:13 |
| pgregory | hi Anteru | 08:22 |
|---|
| Pseudonym | Gotta go. Be back later tonight. | 08:22 |
|---|
| Pseudonym | Night! | 08:22 |
|---|
| pgregory | night | 08:23 |
|---|
| *** Pseudonym has quit IRC | 08:23 |
| Anteru | Morning Paul | 08:24 |
|---|
| *** AlexK has quit IRC | 08:24 |
| pgregory | Anteru: no | 08:24 |
|---|
| *** WayneZacMP has quit IRC | 08:28 |
| Anteru | Em? | 08:30 |
|---|
| pgregory | Anteru: you were about to ask if I'd tested OpenEXR 1.4 on windows yet. | 08:30 |
|---|
| Anteru | Ah, ok - actually I thought it's a kinda late reply at the exception stuff from yesterday ^ :) | 08:31 |
|---|
| Anteru | Bye, gotta go shopping - catch you later | 08:42 |
|---|
| *** Anteru has quit IRC | 08:42 |
| *** AlexK has joined #aqsis | 09:17 |
| *** mafm has joined #aqsis | 09:58 |
| mafm | hi | 09:58 |
|---|
| pgregory | hi mafm, how's things? | 09:59 |
|---|
| mafm | not bad :) | 10:01 |
|---|
| mafm | I have several exams during the next week, so I won't be around, I already told to tcolgate | 10:02 |
|---|
| *** render has joined #aqsis | 10:04 |
| mafm | hi render | 10:09 |
|---|
| render | hi | 10:09 |
|---|
| *** pgregory has quit IRC | 10:27 |
| *** pgregory has joined #aqsis | 10:28 |
| *** ChanServ sets mode: +o pgregory | 10:28 |
| *** pgregory sets mode: +vvv render mafm AlexK | 10:28 |
| pgregory | shame, seems Screen isn't particulalry stable under Cygwin :-( | 10:28 |
|---|
| tcolgate | mornin | 10:32 |
|---|
| tcolgate | pgregory: I don't think I've ever had any problem with it. | 10:32 |
|---|
| pgregory | tcolgate: it just locked up my session | 10:32 |
|---|
| tcolgate | Are you using it with the cygwin console or an xterm? | 10:33 |
|---|
| pgregory | I'm ssh'ing in from work | 10:33 |
|---|
| pgregory | I had irssi running in a Screen session, and it just locked up, I killed the ssh connection, reconnected, and tried to run screen -ls and it just sat there. | 10:34 |
|---|
| tcolgate | oddyou did do a re-attach when you ran it hte second time yeah? | 10:37 |
|---|
| pgregory | I couldn't re-attach, I couldn't even get "screen -ls" to work | 10:38 |
|---|
| tcolgate | ewww | 10:38 |
|---|
| *** Anteru has joined #aqsis | 10:39 |
| Anteru | Re | 10:39 |
|---|
| pgregory | had to login and forcibly kill the screen process | 10:39 |
|---|
| pgregory | I'll give it another go, back shortly, need to restart irssi in a Screen session | 10:41 |
|---|
| Anteru | Hoi Paul | 10:41 |
|---|
| *** pgregory has left #aqsis | 10:41 |
| Anteru | ... | 10:41 |
|---|
| *** pgregory has joined #aqsis | 10:42 |
| *** ChanServ sets mode: +o pgregory | 10:42 |
| pgregory | back | 10:44 |
|---|
| Anteru | wb pgregory | 10:44 |
|---|
| mafm | wb | 10:45 |
|---|
| mafm | I spotted a bit of code which isn't apparently doing anything | 10:45 |
|---|
| mafm | (when trying to identify parts to be parallelized) | 10:45 |
|---|
| pgregory | mafm: where? | 10:45 |
|---|
| mafm | it's the content of #if defined(REQUIRED) | 10:46 |
|---|
| mafm | at 1368, imagebuffer.cpp | 10:46 |
|---|
| mafm | it's apparently counting the number of grids and polygons, but it doesn't do anything with the variables | 10:47 |
|---|
| Anteru | That's maybe a leftover from long time ago when I tried to add some more stats and they turned out to slow down aqsis quite a bit ^ | 10:48 |
|---|
| pgregory | mafm: the #if defined(REQUIRED) masks that code, don't worry about it. It should be removed if it's not needed, that's what we have SVN for, to avoid unnecessary commenting out of code. | 10:49 |
|---|
| mafm | yup | 10:50 |
|---|
| mafm | the thing is that I think that it would be better to remove it from the trunk if possible, and then I would propagate the changes to my branch | 10:51 |
|---|
| pgregory | agreed | 10:51 |
|---|
| mafm | in order to parallelize the code core parts will need to be refactored, the more clean that they are before starting, the better :) | 10:52 |
|---|
| pgregory | 'tis done | 10:53 |
|---|
| * pgregory is waiting on a Boost build here, takes bloody ages. | 10:53 |
| Anteru | 1.34? | 10:54 |
|---|
| tcolgate | Are they still using bjam? | 10:54 |
|---|
| Anteru | Boost.Build2 even ;) | 10:54 |
|---|
| pgregory | "a distributed compile farm, a distributed compile farm, my kingdom for a distributed compile farm" | 10:54 |
|---|
| Anteru | Boost 1.34 x64 took surely one hour here ... | 10:54 |
|---|
| pgregory | yes, and yes. | 10:54 |
|---|
| pgregory | Anteru: i386 is taking 1-2 hours here | 10:54 |
|---|
| mafm | btw, this multithreading project is quite tricky, I (a newcomer) have to mess with the very core of the software :S | 10:54 |
|---|
| Anteru | Hmm ... | 10:54 |
|---|
| tcolgate | mafm: never said it was going to be easy :) | 10:55 |
|---|
| pgregory | and I'm having to do it as an RPM, so I'm learning RPM stuff at the same time, which means each time I screw up the .spec file, I have to rebuild :( | 10:55 |
|---|
| Anteru | .o0 | 10:55 |
|---|
| pgregory | mafm: rest assured, there will be plenty of help along the way, don't be shy, shout when you have trouble. | 10:55 |
|---|
| mafm | :D | 10:55 |
|---|
| pgregory | the people in this room know the code intimately. | 10:56 |
|---|
| Anteru | ^^ Most of them do at least ;) | 10:56 |
|---|
| mafm | I'll need it for sure, thanks | 10:56 |
|---|
| tcolgate | pgregory: Are there no RPMs of the latest boost? | 10:58 |
|---|
| pgregory | not for RHEL4 | 10:58 |
|---|
| mafm | but I was wondering yesterday who'll be changing the most parts of the code: maybe it's other people who has to refactor things instead of me (?) | 10:58 |
|---|
| pgregory | mafm: as you say, it's a very broad project, I'm certain there will be changes that are best left to us. | 10:59 |
|---|
| pgregory | The same applies to Zac (the other SoC student), his code is going to rely on work from other team members. | 10:59 |
|---|
| pgregory | for your project in particular, we need first to decide exactly what you're going to be doing, then we can move on with a plan of action. | 11:00 |
|---|
| Anteru | pgregory: Are there any docs/ideas about exception handling in Aqsis? I just found http://wiki.aqsis.org/dev/code_review?s=exception but the page does not exist yet. | 11:01 |
|---|
| pgregory | not really, we haven't really considered exception handling in Aqsis up to now. | 11:01 |
|---|
| Anteru | If you don't mind I'll think about it a bit ... basically, I'd like to bring in a system similar to .NET/Java and modify the RiAPI generator so no exceptions propagate out of Ri calls. | 11:02 |
|---|
| mafm | btw I'm preparing a kind of Requirements Document in the wiki: http://wiki.aqsis.org/dev/soc_multithreading | 11:03 |
|---|
| pgregory | Anteru: by all means think about it, discuss it, and above all, document it on the Wiki. | 11:04 |
|---|
| mafm | I'm trying to specify now the parts to be made parallel | 11:04 |
|---|
| pgregory | mafm: good, you should notify us via the mailing list when you make major changes to that page. Then we can review the progress. | 11:04 |
|---|
| Anteru | Ok. I just saw it that there was obviously some work in that direction (matrix inverse did report an error if determinant was 0) but it got commented out, so I thought maybe it's time to raise the dead and take a look at it. | 11:05 |
|---|
| Anteru | Moreover, it's a software engineering task and I'm quite comfortable with those :) | 11:05 |
|---|
| mafm | ops, I'm not subscribed to the mailing list r:) | 11:06 |
|---|
| pgregory | sounds good | 11:06 |
|---|
| pgregory | mafm: probably a good idea to subsribe. | 11:06 |
|---|
| tcolgate | The only use of exceptions I can remember in aqsis is buried in the input reading routine of the RIB parser, it's an ugly little bit of code but getting rid of it proved to be non-trbial. | 11:06 |
|---|
| tcolgate | trvial | 11:06 |
|---|
| pgregory | It's not particularly busy at the moment, but I hope the SoC will change that. | 11:06 |
|---|
| pgregory | it == the ml | 11:06 |
|---|
| Anteru | tcolgate: I'd like to have something in place which gives us enough knowledge about the error to treat it right, i.e. whether some invariant was violated (Math error), whether it is system related (I/O, memory) or whether something went really wrong (dice failed or something like that) | 11:07 |
|---|
| *** Joron has joined #Aqsis | 11:08 |
| tcolgate | The problem with the other bit of code is it's not throwing exception for errors really, it's throwing them as part of normal operation (effectively using them to do a long jump). | 11:08 |
|---|
| * pgregory was just looking at the Boost.Wave thing, looks like an ideal candidate to drop the unmaintainable slpp. | 11:09 |
| Anteru | pgregory: Remember when I suggested it ;) | 11:09 |
|---|
| Anteru | tcolgate: Uah?!? You're joking, right? | 11:09 |
|---|
| mafm | subscribed! | 11:09 |
|---|
| mafm | hi Joron | 11:09 |
|---|
| Anteru | Well done, mafm | 11:09 |
|---|
| Joron | Hello all | 11:09 |
|---|
| pgregory | Anteru: yes, and I dismissed it because what we have worked fine. Since then we've discovered a problem with slpp. | 11:09 |
|---|
| tcolgate | We've talked about wave before, the first web page seems clear enough, everything after that seemed to fade into total jibbreish for me. | 11:10 |
|---|
| Anteru | All right ... so what's the deal with Boost.Wave, cause this is something I'm interested in? | 11:10 |
|---|
| Joron | what was the problem with slpp ? | 11:10 |
|---|
| Anteru | Did you discuss it on IRC or somewhere else? | 11:10 |
|---|
| pgregory | we have to write out a temporary file, and the way we handle that at the moment is to use mktemp, which is known to have a race condition risk. | 11:11 |
|---|
| tcolgate | It'd be nice to get something that does gcc style #line XXX filename.cpp stuff again, and that does nested includes properly. | 11:11 |
|---|
| pgregory | we can't use mkstemp, as we need a filename to pass into slpp, not a handle. | 11:11 |
|---|
| Anteru | Yeah, I remember I mentioned Boost.Wave exactly because of that problem. | 11:11 |
|---|
| pgregory | modifying slpp to change the way it works is a 'mare. | 11:12 |
|---|
| * tcolgate nods | 11:12 |
| pgregory | so perhaps now is a good time to consider an alternative. | 11:12 |
|---|
| pgregory | Anteru: pick one thing and stick with it. | 11:12 |
|---|
| pgregory | exception handling is a good thing to do | 11:12 |
|---|
| Anteru | Yeah, I think the exceptions are better now as I don't know how much time I'll be able to spend. | 11:13 |
|---|
| tcolgate | ah yes, now I recall why my brain didn't fit well with wave..... | 11:13 |
|---|
| Anteru | Starting with Boost.Wave and slowing down half-way doesn't help anyone. | 11:13 |
|---|
| tcolgate | it's the bit about the C++ lexer. | 11:13 |
|---|
| tcolgate | It seemed that it is tied (very very tightly) to be a CPP processor for C++ | 11:13 |
|---|
| Anteru | "Both of the included lexers and the library itself are able to act in a C99 compliant mode" | 11:14 |
|---|
| tcolgate | yes, but what will they do if they seem something that isn't C99 either. | 11:15 |
|---|
| Anteru | Is SL really that far away from C99? | 11:16 |
|---|
| tcolgate | Well it's certainly not C. | 11:17 |
|---|
| Anteru | Well, now the users get the full power of variadic macros at hand :) | 11:17 |
|---|
| pgregory | given the Quick Start stuff in the docs, I'm sure it would be easy to put together a quick test and run it over our shaders to see how it behaves. | 11:17 |
|---|
| Anteru | Yeah, and it doesn't look to scary to me. Just beware of the compile time issues ^ | 11:18 |
|---|
| pgregory | what compile time issues? | 11:18 |
|---|
| Anteru | Separation and inclusion compilation models under http://boost.org/libs/wave/doc/compiletime_config.html | 11:18 |
|---|
| Anteru | Catch you later, gotta "go" to work | 11:20 |
|---|
| pgregory | cya | 11:20 |
|---|
| tcolgate | http://www.codeproject.com/cpp/wave_preprocessor.asp?df=100&forumid=14979&select=1116700#xx1116700xx | 11:20 |
|---|
| tcolgate | Which makes sense. | 11:21 |
|---|
| Joron | cya | 11:21 |
|---|
| pgregory | yes, "C-Like" is ambiguous. | 11:21 |
|---|
| tcolgate | So really we "should" implement an SL lexer, or reuse the one we have, or just cave and move down the entire boost route (which I doubt anyone wants to jump into right now). | 11:22 |
|---|
| pgregory | there is a sample supplied that creates a new lexer for the IDL language. | 11:24 |
|---|
| tcolgate | ah, nice | 11:26 |
|---|
| tcolgate | That page does seem rather old | 11:26 |
|---|
| pgregory | I still think the first thing to try is to see how the cpplexer deals with SL. | 11:26 |
|---|
| pgregory | it at all | 11:26 |
|---|
| tcolgate | yeah, it is possible that the terminology on that page is a bit confused and that they mean the pre-processor portions of C99 and C++ rather than the entire language. In which case the C++/C99 code should deal with the pre-processor stuff in SL just fine. | 11:28 |
|---|
| *** Anteru has quit IRC | 11:31 |
| mafm | hmmm | 11:32 |
|---|
| pgregory | gahhh | 11:32 |
|---|
| mafm | in RenderSurfaces there are several calls to RenderMPGs with the same x* and y* parameters, is that normal? | 11:32 |
|---|
| tcolgate | avast! | 11:32 |
|---|
| pgregory | I hate RPM! | 11:33 |
|---|
| pgregory | mafm: hold on... | 11:33 |
|---|
| tcolgate | mafm: any particular cases that have you worried? | 11:33 |
|---|
| tcolgate | pgregory: hating a packaging system really doesn't make much sense. | 11:34 |
|---|
| mafm | there's an initial call that it's always executed, and then it's called in another two places without changing the parameters (unless the method depends on external global variables ?) | 11:34 |
|---|
| pgregory | mafm: yes, that's right. Because during the process of RenderSurfaces, any number of primitives may end up getting diced. We want to 'empty' the waiting MP bucket as soon as possible, because they take a large amount of memory, so we call RenderMPGs at various points in the loop to empty an MP's that may have been inserted in the previous step. | 11:35 |
|---|
| tcolgate | pgregory: hate Jeff Johnson if yo ulike, he's maintained it for years, a lot of people seem to hate him. | 11:35 |
|---|
| pgregory | tcolgate: the problem is, each time it doesn't work, I have to wait another 2 hours. Why can't it build, and retain the build, so I can just to an incremental thing. | 11:35 |
|---|
| pgregory | Instead it cleans everything, unarhives the source and rebuilds the binaries each bleeding time! | 11:36 |
|---|
| mafm | ah all right | 11:36 |
|---|
| mafm | thanks | 11:36 |
|---|
| *** Pseudonym has joined #Aqsis | 11:36 |
| pgregory | mafm: does that make sense. | 11:37 |
|---|
| tcolgate | pgregory: I'm sure you appreciate why it does it that way, even if it's not exactly a great experience. | 11:37 |
|---|
| pgregory | Hi Pseudonym | 11:37 |
|---|
| tcolgate | Pseudonym: hey | 11:37 |
|---|
| *** pgregory sets mode: +vv Joron Pseudonym | 11:37 |
| Pseudonym | G'day., | 11:37 |
|---|
| pgregory | tcolgate: sure, but I'd like to shortcut it while I'm developing the .spec file. | 11:37 |
|---|
| pgregory | life's too bloody short for 2 hour cycles. | 11:38 |
|---|
| mafm | well, from the parallel thingy point of view, I think that we'll have to work a lot :D | 11:38 |
|---|
| tcolgate | mafm:That shouldn't case any problems, the thread is processing the bucket so it can use exactly the same logic. | 11:39 |
|---|
| mafm | in example, there are calls to CurrentBucket all over the place, but you can't rely on that when you have several instances running at the same time | 11:39 |
|---|
| mafm | yep, but you can't have calls to global variables that can change -- or rather you can, but have to lock the access to them | 11:41 |
|---|
| pgregory | mafm: I imagine that code being wrapped up in a class that encapsulates all the "CurrentBucket" status, and implements operator() so can be used in a Boost.Thread. | 11:41 |
|---|
| tcolgate | mafm: I'm not sure I follow, your tread is processing one bucket, so current bucket, for that thread, should return the bucket being procesed. | 11:41 |
|---|
| Pseudonym | Probably should have been done like that in the first place, in retrospect. | 11:42 |
|---|
| tcolgate | CurrentBucket will clearly need to be refactored, for the threaded version it can't be a global returning the same value every time. | 11:42 |
|---|
| mafm | yes, but not only CurrentBucket but all the global variables (or functions having side effects) as well | 11:43 |
|---|
| tcolgate | correct. | 11:44 |
|---|
| pgregory | mafm: yes, all that status will have to be wrapped up into the thread class. | 11:44 |
|---|
| pgregory | mafm: the first task there is to identify all the global status information that needs to be encapsulated. | 11:44 |
|---|
| tcolgate | But then I don't think anyone was expecting this to be a case of just sprinkling a couple of boost thread calls around, we need proper encapsulation of the environment needed for each bit we are going to thread. | 11:44 |
|---|
| mafm | maybe we could try to set the buckets autonomous, processing their own data and not relying on external functionality | 11:46 |
|---|
| mafm | would that be possible? | 11:46 |
|---|
| tcolgate | There are some external resources they will have to rely on, but we'll have to marshall access to those resources. | 11:46 |
|---|
| pgregory | mafm: that pretty much describes the process. | 11:47 |
|---|
| tcolgate | Like the texture cache. for instance, but that shouldn't actually have too many issues. | 11:47 |
|---|
| Joron | byebye | 11:49 |
|---|
| *** Joron has quit IRC | 11:49 |
| Pseudonym | Those resources can be monitors to begin with. | 11:49 |
|---|
| Pseudonym | Although the texture cache is logically read-only, so that's not a good long-term solution. | 11:50 |
|---|
| Pseudonym | One useful trick, by the way, is that hash tables can easily support quite fine-grained locking. | 11:50 |
|---|
| Pseudonym | Because hash lines can be locked individually. | 11:51 |
|---|
| Pseudonym | Trees, for example, are much harder to do., | 11:51 |
|---|
| mafm | well, if it's read-only then there are no problems | 11:53 |
|---|
| Pseudonym | No, it's _logically_ read-only. | 11:53 |
|---|
| Pseudonym | In practice, it's a cache. | 11:54 |
|---|
| Pseudonym | So elements are going to enter and leave. | 11:54 |
|---|
| mafm | oh, right | 11:55 |
|---|
| tcolgate | I know there is some work being done on the texture cache at the moment, but you should probably ignore that for now. | 11:56 |
|---|
| Pseudonym | Yeah. WHo's doing that? Zac? | 11:56 |
|---|
| tcolgate | pgregory: Before I forget, how often does our matrix inversion code get hit do you think? Anteru and I swapped a few ideas on possible ways to speed it up, not sure if it'd be worth it though. | 11:57 |
|---|
| tcolgate | atleast at ht emoment. | 11:57 |
|---|
| pgregory | the matrix inversion code is used a lot IIRC | 11:57 |
|---|
| tcolgate | Pseudonym: Nope, Chris I think | 11:58 |
|---|
| Pseudonym | What did you come up with? | 11:58 |
|---|
| Pseudonym | Right. | 11:58 |
|---|
| tcolgate | two things.... | 11:58 |
|---|
| pgregory | tcolgate: it is also the cause of a bug at the moment, but I don't know the right solution. | 11:58 |
|---|
| tcolgate | I read something that said that for the case of pure rotation or translation matrices there are fast methods for doing inversion, so we could possibly test for those cases. | 11:58 |
|---|
| pgregory | the problem is, it's impossible to invert a matrix that has a projection. | 11:58 |
|---|
| pgregory | Pseudonym: tcolgate is right, Chris is doing the texture refactor, Zac is doing DSM. | 11:59 |
|---|
| Pseudonym | Right. | 11:59 |
|---|
| tcolgate | Seconds, I thought we could cache the result and set up a pointer to the inversion of a matrix if we have already created it (and have that inversoin contain a back reference to the matrix it was an inversion of in the same way. | 11:59 |
|---|
| Pseudonym | Gotcha. | 11:59 |
|---|
| Pseudonym | tcolgate: I tried the second once, and it's tricky. | 11:59 |
|---|
| Pseudonym | Because if you invert the inversion... | 11:59 |
|---|
| Pseudonym | It also makes the structure more heavy-weight, since it has a pointer in it. | 11:59 |
|---|
| Pseudonym | But the first idea is very doable. | 12:00 |
|---|
| Pseudonym | The trick is that you keep track of the history of the matrix in a small bitvector. | 12:00 |
|---|
| Pseudonym | (Less than 8 bits, IIRC.) | 12:00 |
|---|
| tcolgate | yes, I'm showing my naivetie here, but I assumed that the inverse of the inversion would be the original. | 12:00 |
|---|
| Pseudonym | The bitvector represents the subgroups that the matrix belongs to. | 12:01 |
|---|
| Pseudonym | Paul: BTW, inverting a projection matrix is just inverting the 4x4 matrix. | 12:01 |
|---|
| Pseudonym | Semantically, that's the right thing to do. | 12:01 |
|---|
| Pseudonym | It doesn't make a lot of sense, but that's wha tyou do,. | 12:01 |
|---|
| pgregory | Pseudonym: well, it doesn't work for us, we get #NaN's | 12:01 |
|---|
| Pseudonym | What should it return? | 12:02 |
|---|
| Pseudonym | Or should it throw? | 12:02 |
|---|
| pgregory | Pseudonym: it probably shouldn't return anything, but the problem is, we need it. | 12:02 |
|---|
| tcolgate | Well I suppose NaN is as good a thing as any for it to return. | 12:03 |
|---|
| mafm | hmmm, is there any reason for the CqBucket having so many static stuff? | 12:03 |
|---|
| Pseudonym | mafm: Apart from statistics, no. | 12:04 |
|---|
| Pseudonym | Hysterical raisins. | 12:04 |
|---|
| pgregory | This is the bug... | 12:04 |
|---|
| mafm | I mean this: | 12:05 |
|---|
| pgregory | http://sourceforge.net/tracker/index.php?func=detail&aid=1696939&group_id=25264&atid=383970 | 12:05 |
|---|
| mafm | private: | 12:05 |
|---|
| mafm | static TqInt m_XOrigin; ///< Origin in discrete coordinates of this bucket. | 12:05 |
|---|
| mafm | static TqInt m_YOrigin; ///< Origin in discrete coordinates of this bucket. | 12:05 |
|---|
| pgregory | the problem is, for all transformations, we go 'through' "world", but we can't go through "world" from "NDC", or "screen" | 12:05 |
|---|
| pgregory | mafm: the reason that is all static is that there currently is only ever one bucket being processed. | 12:06 |
|---|
| Pseudonym | Ah. | 12:06 |
|---|
| mafm | ok | 12:06 |
|---|
| pgregory | mafm: the 'non-static' stuff is the stuff that needs to be around for each bucket, i.e. the primitives/MP lists, that can be populated before we reach the bucket for processing. | 12:06 |
|---|
| tcolgate | >>>>> (11:52:32 AM) Pseudonym: Hysterical raisins. | 12:06 |
|---|
| Pseudonym | Paul: That suggests that it's not the inversion code that's the problem, it's going through "world". | 12:06 |
|---|
| tcolgate | #I look away for five seconds and it al mental in here! | 12:07 |
|---|
| Pseudonym | :-) | 12:07 |
|---|
| pgregory | Pseudonym: yes, but that's pretty deeply ingrained in our code. | 12:07 |
|---|
| Pseudonym | tcolgate: google it | 12:07 |
|---|
| pgregory | mafm: does that make sense? | 12:07 |
|---|
| pgregory | mafm: basically, all the stuff that can 'exist' for any bucket is non-static, the stuff that we only care about for the currently being processes bucket is static, and changes to match the currently processed bucket. | 12:08 |
|---|
| pgregory | mafm: it just avoids having tonnes of useless information around in the bucket array. | 12:08 |
|---|
| tcolgate | Pseudonym: nice, haven't heard that one before. | 12:10 |
|---|
| pgregory | mafm: bear in mind an HD image at a bucket size of 16x16 would need 8100 buckets. | 12:10 |
|---|
| Pseudonym | tcolgate: When I was an undergrad, I read the jargon file a lot. | 12:10 |
|---|
| mafm | so probably all that code would have to be made non-static, but that brings us to a very high memory footprint... | 12:10 |
|---|
| Pseudonym | Hang on, BRB. Need to take kiddies to bed. | 12:11 |
|---|
| pgregory | mafm: not necessarily | 12:12 |
|---|
| mafm | do the buckets really need to be instantiated during all the process? they can be created on the fly if they're independent from each other | 12:12 |
|---|
| pgregory | mafm: you could keep the current system for managing the bucket contents, then each time you allocate a bucket to a thread, you introduce a new class that has all the other data, and references the bucket from the array. | 12:13 |
|---|
| Pseudonym | So a "bucket" could actually belong to a worker thread. | 12:13 |
|---|
| pgregory | mafm: all buckets need to be around from the start, as it's entirely possible that a primitive could be placed into the last bucket in the image from the off. | 12:14 |
|---|
| tcolgate | We need a scheduler for allocating buckets to threads. | 12:14 |
|---|
| pgregory | large primitives can easily sit in many buckets, waiting to be processed. | 12:14 |
|---|
| tcolgate | It is also possible that processing of one bucket in another thread could add content to the bucket you are currenly processing in a different thread. | 12:14 |
|---|
| Pseudonym | Yeah, I've been thinking about that one a bit. | 12:15 |
|---|
| Pseudonym | How does a bucket know when it's actually finished? | 12:15 |
|---|
| pgregory | Pseudonym: when the bucket processing list passes it. | 12:15 |
|---|
| * Pseudonym nods | 12:15 |
| Pseudonym | Ah, good point | 12:16 |
|---|
| pgregory | nothing can be placed in 'previous' buckets. | 12:16 |
|---|
| Pseudonym | I thought of another option, but I don't know if it's a good idea or not. | 12:16 |
|---|
| Pseudonym | WHen you add a piece of geometry, you increment a reference count in all buckets that it might touch. | 12:16 |
|---|
| Pseudonym | When you split a bucket, you increment the count for the child geometry and then decrement for the parent geometry. (The order is important!) | 12:17 |
|---|
| Pseudonym | The bucket is done when the counter reaches zero. | 12:17 |
|---|
| * mafm reading | 12:17 |
| tcolgate | hmmmm.... | 12:18 |
|---|
| tcolgate | bucket ordering... | 12:18 |
|---|
| Pseudonym | I'm not certain if it's a good idea because the counters could easily become a bottleneck. | 12:18 |
|---|
| * mafm getting a headache :S :) | 12:18 |
| tcolgate | We've done a bit of thinking before out allowing different ordering for buckte processing, clearly that's an issue here, unless Andres suggestion can work. | 12:19 |
|---|
| tcolgate | mafm: You've not even met the shader vm yet :) | 12:19 |
|---|
| Pseudonym | Well, you still need to process the buckets more or less in order. | 12:19 |
|---|
| Pseudonym | Because large surfaces will still start off life in the top left-most bucket. | 12:19 |
|---|
| mafm | I'm not sure I want to met him :P | 12:20 |
|---|
| tcolgate | First off, is bucket order an issue, do any of the other renderers allow for different bucket ordering? | 12:20 |
|---|
| Pseudonym | I don't think so. | 12:20 |
|---|
| Pseudonym | It's simply not a useful control. | 12:20 |
|---|
| Pseudonym | If you care, render it piecemeal. | 12:21 |
|---|
| tcolgate | If not then we can just make an executive decision on it, Paul, do we care about allowing different bucket processing orders? | 12:21 |
|---|
| pgregory | tcolgate: different bucket ordering can be useful, and yes, other renderers do support some bucket ordering choice. | 12:22 |
|---|
| mafm | I'm going to lunch now, be back in a while | 12:22 |
|---|
| Pseudonym | Such as? | 12:22 |
|---|
| pgregory | the classic example... | 12:22 |
|---|
| tcolgate | At one point I did try and come up with a process that was intended to maximise localtisation of rendering to improve texture cache coherencey, but I could never quite get it right in my head. | 12:22 |
|---|
| pgregory | you have a LotR style battle sequence, with an army or 100,000 orcs. The camera is looking out over the hoards. the render starts fine, then as soon as it hits the first line, it barfs as it has to load the entire battallion. | 12:23 |
|---|
| pgregory | if you render in columns, you can get better performance out of your caching in that situation. | 12:23 |
|---|
| Pseudonym | I would have thought that DelayedReadArchive would pretty much fix it, too. | 12:24 |
|---|
| Pseudonym | So who supports that? | 12:24 |
|---|
| pgregory | Pseudonym: no, as the DRA will have to load for the entire row. | 12:24 |
|---|
| Pseudonym | Ah, OK. | 12:24 |
|---|
| Pseudonym | And apart from row vs column, does anyone support any other orderings? | 12:25 |
|---|
| Pseudonym | If those are the only two possibilities, that should be doable. | 12:25 |
|---|
| pgregory | I know WETA rendered the LotR battle stuff on it's side and rotated it in post for that very reason. | 12:26 |
|---|
| pgregory | Pseudonym: 3Delight supports "horizontal", "zigzag", "vertical", "spiral", "circle" and "random". With a caveat on "random" ["Should not be used since it is not memory efficient"] | 12:28 |
|---|
| Pseudonym | Yow. | 12:28 |
|---|
| Pseudonym | Some of those sound more useful than others. | 12:28 |
|---|
| pgregory | *cough* gimmick! *cough* | 12:28 |
|---|
| Pseudonym | Yeah. | 12:28 |
|---|
| Pseudonym | Horizontal and vertical make the msot sense. | 12:29 |
|---|
| Pseudonym | Zigzag... well, if you're producing jpegs directly, I guess. | 12:29 |
|---|
| Pseudonym | Growing outwards from a point of interest I can see. | 12:29 |
|---|
| pgregory | I think other renderers support things like spiral to make processing of GI data more efficient, and to allow easier preview, you can tell a lot sooner that your render is rubbish if yo usee it from the middle out. | 12:30 |
|---|
| * Pseudonym nods | 12:30 |
| pgregory | I guess the only way we can be sure is to come up with a set of 'rules' for each method, that basically encapsulate the following query... | 12:31 |
|---|
| pgregory | "given the current list of completed buckets, can <this> bucket accept any more data?" | 12:31 |
|---|
| Pseudonym | Hmm. | 12:31 |
|---|
| Pseudonym | If we're looking at 10,000 or so buckets, then precomputing the bucket order and just scanning through it is feasable. | 12:32 |
|---|
| pgregory | ? | 12:32 |
|---|
| Pseudonym | OK, think about this for a moment. | 12:32 |
|---|
| Pseudonym | Suppose we use my idea of putting a counter in each bucket. | 12:32 |
|---|
| Pseudonym | So for each surface that we add, we're sold on the fact that we're going to scan every bucket that it touches. | 12:33 |
|---|
| Pseudonym | With me? | 12:33 |
|---|
| pgregory | yep | 12:33 |
|---|
| Pseudonym | OK. | 12:33 |
|---|
| Pseudonym | Before we start reading in geometry, we effectively sort the buckets by the order that they're going to be visited in. | 12:34 |
|---|
| Pseudonym | And we represent this with an integer in each bucket. | 12:34 |
|---|
| Pseudonym | Now. | 12:35 |
|---|
| Pseudonym | When we add a piece of geometry to the scene, we add it to the bucket with the smallest "number" that isn't already finished. | 12:35 |
|---|
| Pseudonym | Since we scan all of the buckets that it touches, we can find out which one it is as we go. | 12:35 |
|---|
| pgregory | right, that basically mimics what we currently do, but more flexibly. | 12:36 |
|---|
| Pseudonym | Then when we've finished scanning, pop it in that bucket. | 12:36 |
|---|
| Pseudonym | Right. | 12:36 |
|---|
| Pseudonym | And with minimal overhead. | 12:36 |
|---|
| pgregory | how does that help us determine if a bucket is 'really' finished when it is empty? | 12:36 |
|---|
| Pseudonym | Ah. | 12:36 |
|---|
| Pseudonym | OK. | 12:36 |
|---|
| Pseudonym | When we add a piece of geometry, we first scan all of the buckets that it touches. | 12:37 |
|---|
| Pseudonym | As we do that, we increment a reference count in the bucket. | 12:37 |
|---|
| pgregory | right, with you | 12:37 |
|---|
| Pseudonym | A nonzero reference count means "there's still some more geometry to go in here". | 12:37 |
|---|
| pgregory | "there *might* still be some more geometry to go in here" | 12:38 |
|---|
| pgregory | as the bound is aligned | 12:38 |
|---|
| Pseudonym | When we split a surface, we increment the reference counts for the buckets that the children touch, THEN decrement the reference counts for the parent. | 12:38 |
|---|
| Pseudonym | The order is important. | 12:38 |
|---|
| pgregory | that would mean a bucket needs references to it's geometry, *and* the geometry needs references to it's bucket(s). | 12:39 |
|---|
| Pseudonym | No! | 12:39 |
|---|
| Pseudonym | The geometry only needs to know its bounding box in bucket coordinates. | 12:39 |
|---|
| pgregory | how do we decrement the count then? we scan again? isn't that a bit costly | 12:39 |
|---|
| Pseudonym | It might be. | 12:40 |
|---|
| pgregory | given how many times we split geometry. | 12:40 |
|---|
| Pseudonym | True, but remember that we're going to spend very little time manipulating reference counts compared to, say, evaluating shaders. | 12:40 |
|---|
| Pseudonym | You're right, it might cost. | 12:41 |
|---|
| pgregory | yeah, and it could be optimised later if it showed up as a bottleneck. | 12:41 |
|---|
| * Pseudonym nods | 12:41 |
| Pseudonym | THere are techniques for optimising this. | 12:41 |
|---|
| pgregory | sure | 12:41 |
|---|
| pgregory | that sounds like a very workable solution, and will afford us the flexibility to choose completely arbitrary rendering orders. | 12:42 |
|---|
| Pseudonym | One thing that springs to mind, for example, is that you don't need the reference count for a bucket until the moment that you want to start processing it. | 12:42 |
|---|
| * pgregory notes we have another great idea on the table, the table is getting quite full. | 12:42 |
| pgregory | we need some people around the table grabbing the ideas ;-) | 12:43 |
|---|
| Pseudonym | :-) | 12:43 |
|---|
| Pseudonym | I'm looking at the CoordSysTransform problem at the moment. | 12:44 |
|---|
| pgregory | excellent, that would be a good one to clean up. | 12:44 |
|---|
| Pseudonym | Maybe everywhere that we store world space, we should store screen space right next to it. | 12:44 |
|---|
| pgregory | I'd like to get the new FB to a 'workable' stage soon, then merge that in for others to play with. | 12:44 |
|---|
| pgregory | then I will be free to take on some other task. | 12:45 |
|---|
| * Pseudonym nods | 12:45 |
| Pseudonym | Oh, back to the matrix inversion thing. | 12:45 |
|---|
| Pseudonym | The idea, and it's from one of the Graphics Gems books, is to keep track of what subgroups a transformation belongs to. | 12:45 |
|---|
| pgregory | yeah, we used to do that at Superscape, showed massive wins for us. | 12:46 |
|---|
| Pseudonym | So, for example, a rotation is distance and angle preserving, and it's orthonormal. | 12:46 |
|---|
| Pseudonym | Yeah. | 12:46 |
|---|
| pgregory | you can tailor things like matrix multiplies using that information too. | 12:46 |
|---|
| Pseudonym | And when you multiply two matrices, you just "and" the bit vectors. | 12:46 |
|---|
| Pseudonym | That'd be a fun task for a beginner, I think. | 12:46 |
|---|
| * pgregory thinks there are as many definitions of the word "fun" as there are people in this chatroom. :) | 12:47 |
| Pseudonym | At the moment, we keep track of whether or not a matrix is an identity. | 12:47 |
|---|
| Pseudonym | "Fun" in this sense means "easy to do, not requiring a lot of experience with the code base, and likely to show benefits quickly". | 12:47 |
|---|
| pgregory | agreed | 12:48 |
|---|
| pgregory | we should add that to the list of tasks on the Wiki | 12:48 |
|---|
| tcolgate | sorry, stuck in the real world for a bit | 12:48 |
|---|
| pgregory | tcolgate: I think it'll take a while to catch up, I've never seen this forum so active. | 12:49 |
|---|
| tcolgate | God damn police forces | 12:50 |
|---|
| pgregory | ? | 12:50 |
|---|
| Pseudonym | Sounds serious. | 12:50 |
|---|
| pgregory | probably just means the UK police force can't send emails at the moment :-) | 12:50 |
|---|
| *** scorp007 has joined #aqsis | 12:51 |
| pgregory | hi scorp007 | 12:51 |
|---|
| *** pgregory sets mode: +v scorp007 | 12:51 |
| scorp007 | hi all | 12:52 |
|---|
| pgregory | Pseudonym: would you mind detailing your "out of order bucket processing" proposal on the Wiki? | 12:52 |
|---|
| tcolgate | ooh blimey, all very productive | 12:52 |
|---|
| tcolgate | Ok, so re: bucket ordering. | 12:52 |
|---|
| Pseudonym | Erm... OK. | 12:52 |
|---|
| pgregory | cheerws | 12:52 |
|---|
| tcolgate | Our bucket scheduler I suppose effectively becomes the things that gives the ordering, and for any given scheduler we could just have a method that determines the completion of buckets too. | 12:53 |
|---|
| tcolgate | I need to re-read Andrews stuff to get my heard round it though, as perhaps that's a more generic solution. | 12:54 |
|---|
| Pseudonym | There might be, yes. | 12:54 |
|---|
| pgregory | tcolgate: it's totally generic | 12:54 |
|---|
| tcolgate | re: matrix math, I've got the GG series lying around at home, perhaps I should have a trawl through them. | 12:54 |
|---|
| pgregory | although, as with 3Delight, certain bucket orders will be more efficient than others. | 12:54 |
|---|
| Pseudonym | tcolgate: Yes! | 12:54 |
|---|
| Pseudonym | Please do. | 12:54 |
|---|
| Pseudonym | I think that "random" makes no sense. | 12:55 |
|---|
| Pseudonym | Something more like "Peano" might. | 12:55 |
|---|
| * Pseudonym rebuilds... just touched renderer.h... | 12:55 |
| * Pseudonym waits forever... | 12:55 |
| tcolgate | I was trying to figure out a strategy that would basically result in an ordering similar to the one used in jpeg compression | 12:55 |
|---|
| Pseudonym | tcolgate: See above, re "zigzag". | 12:56 |
|---|
| * pgregory notes, that's another thing we need to review, header inclusions. | 12:56 |
| tcolgate | I didn't want to be too literal on the zig zag thing though, I basically wanted to follow where the geometry we had created had gone to where possible. | 12:57 |
|---|
| pgregory | "peano", "hilbert", the list is endless | 12:57 |
|---|
| tcolgate | peano sounds like Pinot.... mmmmmmm...... wine. | 12:57 |
|---|
| Pseudonym | I guess you kind of want something pseudo-random. | 12:58 |
|---|
| Pseudonym | But surely "large prime" would do. | 12:58 |
|---|
| tcolgate | The idea was that if, when we are splitting and dicing, we push loads of geometry into the ucket below us, then we move on to that one next. | 12:59 |
|---|
| tcolgate | then we'd move back up and right later. | 12:59 |
|---|
| pgregory | hmm, that wouldn't fit into the proposal. | 12:59 |
|---|
| pgregory | the proposal relies (unless I'm mistaken) on an identifiable ordering up front. | 13:00 |
|---|
| tcolgate | THe idea is to keep rendering in busy areas for as long as we can on the assumption that you'd have some locality of texture access. | 13:01 |
|---|
| Pseudonym | Paul, just checking one thing. | 13:01 |
|---|
| tcolgate | pgregory: header inclusion really should be part of the code review. | 13:01 |
|---|
| Pseudonym | Is it just the geometry projection that's a problem with this NDC space thing, or are there problems with the shader too? | 13:02 |
|---|
| scorp007 | whoa, I have't read all the logs, but I'd just like to add that in addition to the bucket orders listed already, some programs provide interactive bucket ordering as a temporary override to the default. So if a user really wants to see a part of an image rendered first, that area will be given priority, then the rendering will continue as normal. | 13:02 |
|---|
| scorp007 | Mantra and Vray implement this | 13:02 |
|---|
| scorp007 | but that's just bells and whistles, I guess | 13:03 |
|---|
| pgregory | it's anything that wants to use the CqRenderer matSpaceToSpace call where the "from" is "NDC" or 'later'. | 13:03 |
|---|
| pgregory | scorp007: yes, as long as the order is defineable up front, it should be possible to support it. | 13:03 |
|---|
| scorp007 | when you say up front, do you mean at the beginning of the rendeR? | 13:04 |
|---|
| pgregory | scorp007: yes | 13:04 |
|---|
| scorp007 | mantra and vray do it at runtime, via events to the framebuffer | 13:04 |
|---|
| tcolgate | scorp007: For a ray tracer that is easy enough to do, harder in REYEs though. | 13:05 |
|---|
| pgregory | scorp007: we can't easily support that. | 13:05 |
|---|
| scorp007 | tcolgate: I guess that's true | 13:05 |
|---|
| scorp007 | pgregory: sure, it's not something I consider vital | 13:05 |
|---|
| scorp007 | just thought I'd mention it | 13:05 |
|---|
| tcolgate | Also, as long as we stick to the standard display driver interface, you can't really give that kind of user feedback to the renderer. | 13:05 |
|---|
| scorp007 | well, exhibit provides a two-way communication, doesn't it? | 13:06 |
|---|
| tcolgate | We could possibly support something along those lines for re-rendering if we let eqshibit launch aqsis with some crop window value I suppose. | 13:06 |
|---|
| scorp007 | so you can perhaps pause/cancel the render via the framebuffer? | 13:06 |
|---|
| scorp007 | that would indeed be useful | 13:06 |
|---|
| pgregory | scorp007: not currently, it's all one way. | 13:07 |
|---|
| scorp007 |
|---|